2024-03-19T02:26:04.774 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T02:26:04.774 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441 --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/orig.config.yaml 2024-03-19T02:26:04.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T02:26:04.922 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441 branch: wip-batrick-testing-20240318.181317 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609441' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFx719IkU9FJAu59YtRWy7dpjI33NS5tXulL6SGSomLxamg9kPlqlUVUqWY0X+kfcWkNNlTabPYR7Zi+Zj+FNo= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDa8khp7bshqgwJj1KEBwKqaixzbd4GAw47cz1EWzErQ2TUConfFiFw4/FY8kSd10LsmnQXHr5Lw0wyvVfMeYkA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T02:26:04.922 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T02:26:04.923 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T02:26:04.923 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T02:26:04.925 INFO:teuthology.task.internal:Checking packages... 2024-03-19T02:26:04.948 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T02:26:04.948 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:26:04.948 INFO:teuthology.packaging:ref: None 2024-03-19T02:26:04.949 INFO:teuthology.packaging:tag: None 2024-03-19T02:26:04.949 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T02:26:04.949 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:26:04.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T02:26:05.176 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T02:26:05.178 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T02:26:05.190 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T02:26:05.190 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T02:26:05.236 INFO:teuthology.task.internal:Saving configuration 2024-03-19T02:26:05.258 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T02:26:05.273 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:26:05.302 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.858958', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFx719IkU9FJAu59YtRWy7dpjI33NS5tXulL6SGSomLxamg9kPlqlUVUqWY0X+kfcWkNNlTabPYR7Zi+Zj+FNo='} 2024-03-19T02:26:05.323 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.860582', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDa8khp7bshqgwJj1KEBwKqaixzbd4GAw47cz1EWzErQ2TUConfFiFw4/FY8kSd10LsmnQXHr5Lw0wyvVfMeYkA='} 2024-03-19T02:26:05.323 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T02:26:05.333 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-19T02:26:05.333 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-19T02:26:05.333 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T02:26:05.432 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f125ebc0160>, signals=[15]) 2024-03-19T02:26:05.433 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T02:26:05.445 INFO:teuthology.task.internal:Opening connections... 2024-03-19T02:26:05.445 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:26:05.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:26:05.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:26:05.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:26:05.678 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T02:26:05.685 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-03-19T02:26:05.702 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-03-19T02:26:05.703 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-03-19T02:26:05.759 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-03-19T02:26:05.759 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-03-19T02:26:05.759 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-03-19T02:26:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:26:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:26:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:26:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:26:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:26:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:26:05.761 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-03-19T02:26:05.784 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-03-19T02:26:05.801 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-03-19T02:26:05.802 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:26:05.859 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:26:05.860 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:26:05.860 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-03-19T02:26:05.883 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T02:26:05.894 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T02:26:05.925 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T02:26:05.925 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:26:05.928 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:26:05.943 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T02:26:05.958 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T02:26:05.958 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:26:05.984 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:26:06.246 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T02:26:06.307 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T02:26:06.323 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-19T02:26:06.323 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-19T02:26:06.323 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-19T02:26:06.323 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:26:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:26:06.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-19T02:26:06.744 DEBUG:teuthology.task.kernel:sha1 for client.0 is 908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:26:06.744 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "908b6b276e8f14e16032b32a9770077cd37b6bd4"... 2024-03-19T02:26:06.744 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-19T02:26:06.761 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:26:06.761 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:26:06.761 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-19T02:26:06.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum upgrade -y linux-firmware 2024-03-19T02:26:08.951 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-19T02:26:10.552 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-19T02:26:12.667 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-19T02:30:48.160 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 70 kB/s | 19 MB 04:34 2024-03-19T02:30:54.551 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-19T02:30:56.226 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 632 kB/s | 21 MB 00:33 2024-03-19T02:31:34.812 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-19T02:31:37.025 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-19T02:31:37.037 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-19T02:31:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-19T02:31:37.037 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-19T02:31:37.037 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-03-19T02:31:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 1 Package 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 375 M 2024-03-19T02:31:37.038 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-19T02:31:44.154 INFO:teuthology.orchestra.run.smithi092.stdout:linux-firmware-20240219-143.el9.noarch.rpm 53 MB/s | 375 MB 00:07 2024-03-19T02:31:44.156 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:31:44.156 INFO:teuthology.orchestra.run.smithi092.stdout:Total 53 MB/s | 375 MB 00:07 2024-03-19T02:31:44.156 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-19T02:31:44.265 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-19T02:31:44.265 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-19T02:31:46.700 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-19T02:31:46.701 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-19T02:31:52.908 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-19T02:31:53.146 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-19T02:31:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:31:54.510 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:31:54.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-19T02:31:54.903 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:31:54.904 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:31:54.904 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-03-19T02:31:54.904 INFO:teuthology.orchestra.run.smithi092.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-19T02:31:54.904 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:31:54.904 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-19T02:31:55.202 INFO:teuthology.task.kernel:Downloading kernel 908b6b276e8f14e16032b32a9770077cd37b6bd4 on client.0... 2024-03-19T02:31:55.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:31:55.344 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/ 2024-03-19T02:31:55.345 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g908b6b276e8f-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-19T02:31:55.347 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi092.front.sepia.ceph.com to complete... 2024-03-19T02:32:16.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19 02:32:16 URL:https://1.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/kernel-6.8.0_g908b6b276e8f-1.x86_64.rpm [336334597/336334597] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-19T02:32:16.395 INFO:teuthology.orchestra.run.smithi092.stderr:FINISHED --2024-03-19 02:32:16-- 2024-03-19T02:32:16.395 INFO:teuthology.orchestra.run.smithi092.stderr:Total wall clock time: 21s 2024-03-19T02:32:16.396 INFO:teuthology.orchestra.run.smithi092.stderr:Downloaded: 1 files, 321M in 21s (15.5 MB/s) 2024-03-19T02:32:16.398 INFO:teuthology.task.kernel:Installing kernel 908b6b276e8f14e16032b32a9770077cd37b6bd4 on client.0... 2024-03-19T02:32:16.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-19T02:32:18.329 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying... ######################################## 2024-03-19T02:32:18.385 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing... ######################################## 2024-03-19T02:32:18.465 INFO:teuthology.orchestra.run.smithi092.stdout:Updating / installing... 2024-03-19T02:32:26.105 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-6.8.0_g908b6b276e8f-1 ######################################## 2024-03-19T02:32:42.175 INFO:teuthology.orchestra.run.smithi092.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:33:04.692 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-19T02:33:04.692 DEBUG:teuthology.orchestra.run.smithi092:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/System.map 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/config 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-19T02:33:04.793 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-19T02:33:04.794 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu/mce 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/ansi_cprng.ko 2024-03-19T02:33:04.795 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/anubis.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/arc4.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_memcpy.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_pq.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_tx.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_xor.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/raid6test.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/authenc.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/authencesn.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blake2b_generic.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blowfish_common.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blowfish_generic.ko 2024-03-19T02:33:04.796 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/camellia_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast5_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast6_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast_common.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/ccm.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cmac.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crc32_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crypto_engine.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crypto_user.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cts.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/des_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/dh_generic.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/echainiv.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/essiv.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/fcrypt.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/khazad.ko 2024-03-19T02:33:04.797 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/lz4.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/lz4hc.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/md4.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/michael_mic.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/pcbc.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/pcrypt.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/rmd160.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/seed.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/serpent_generic.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/tcrypt.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/tea.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/twofish_common.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/twofish_generic.ko 2024-03-19T02:33:04.798 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/vmac.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/wp512.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xcbc.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xor.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xxhash_generic.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/acpi_pad.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/custom_method.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/ec_sys.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/platform_profile.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/sbs.ko 2024-03-19T02:33:04.799 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/sbshc.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/video.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/acard-ahci.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/ahci_platform.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/ata_generic.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/libahci_platform.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_acpi.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ali.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_amd.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_artop.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_atiixp.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_atp867x.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cmd640.ko 2024-03-19T02:33:04.800 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cmd64x.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cypress.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_efar.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt366.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt37x.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_it8213.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_it821x.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_jmicron.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_marvell.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_mpiix.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_netcell.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ninja32.ko 2024-03-19T02:33:04.801 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ns87410.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ns87415.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_oldpiix.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_opti.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_optidma.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pcmcia.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_piccolo.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_rdc.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sch.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_serverworks.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sil680.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sis.ko 2024-03-19T02:33:04.802 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sl82c105.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_triflex.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_via.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pdc_adma.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_inic162x.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_mv.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_nv.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_promise.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_qstor.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sil.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sil24.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sis.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_svw.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sx4.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_uli.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_via.ko 2024-03-19T02:33:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_vsc.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/auxdisplay 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/auxdisplay/ks0108.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/bcma 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/bcma/bcma.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/aoe 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/aoe/aoe.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/brd.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/drbd 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/drbd/drbd.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/floppy.ko 2024-03-19T02:33:04.804 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/loop.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/mtip32xx 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/nbd.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/null_blk 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/null_blk/null_blk.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/pktcdvd.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/rbd.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/virtio_blk.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkback 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkfront.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/zram 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/zram/zram.ko 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char 2024-03-19T02:33:04.805 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hangcheck-timer.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/via-rng.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-19T02:33:04.806 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/lp.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/mwave 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/mwave/mwave.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ppdev.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tlclk.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/uv_mmtimer.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/virtio_console.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-19T02:33:04.807 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp/ccp.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_common 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-19T02:33:04.808 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/padlock-aes.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/padlock-sha.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/virtio 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dca 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dca/dca.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/devfreq 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/dw 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/ioat 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac 2024-03-19T02:33:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/amd64_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/e752x_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/edac_mce_amd.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i3000_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i3200_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i5100_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i5400_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i7300_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i7core_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i82975x_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/ie31200_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/sb_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/x38_edac.ko 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware 2024-03-19T02:33:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware/edd.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio/gpio-ich.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-appleir.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-aureal.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-axff.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-dr.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-elecom.ko 2024-03-19T02:33:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-elo.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-emsff.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gaff.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gt683r.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gyration.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtekff.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-hyperv.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-icade.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-keytouch.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-kye.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-lcpower.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-led.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-lenovo.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-19T02:33:04.812 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-multitouch.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-ortek.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-penmount.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-petalynx.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-picolcd.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-pl.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-plantronics.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-primax.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-rmi.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-common.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-19T02:33:04.813 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-saitek.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-samsung.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sjoy.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sony.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-speedlink.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-steelseries.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sunplus.ko 2024-03-19T02:33:04.814 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-tivo.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-tmff.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-topseed.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-twinhan.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-uclogic.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-waltop.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-wiimote.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-xinmo.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-zpff.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-zydacron.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/uhid.ko 2024-03-19T02:33:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/wacom.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_balloon.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_utils.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_vmbus.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/abituguru.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/abituguru3.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ad7414.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ad7418.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adc128d818.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1025.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1026.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1029.ko 2024-03-19T02:33:04.816 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1031.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm9240.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ads7828.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7410.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7411.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7462.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7470.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7475.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7x10.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/amc6821.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/applesmc.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asb100.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asc7621.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-19T02:33:04.817 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/atxp1.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/coretemp.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/dme1737.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ds1621.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ds620.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/emc1403.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/emc6w201.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f71805f.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f71882fg.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f75375s.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/fam15h_power.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/fschmd.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/g760a.ko 2024-03-19T02:33:04.818 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/g762.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/gl518sm.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/gl520sm.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/i5500_temp.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/i5k_amb.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ibmaem.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ibmpex.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ina209.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ina2xx.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/it87.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/k10temp.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/k8temp.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lineage-pem.ko 2024-03-19T02:33:04.819 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm63.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm73.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm75.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm77.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm78.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm80.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm83.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm85.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm87.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm90.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm92.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm93.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95234.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95241.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95245.ko 2024-03-19T02:33:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc2945.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4151.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4215.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4222.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4245.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4260.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4261.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max16065.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max1619.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max1668.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max197.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6639.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6650.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6697.ko 2024-03-19T02:33:04.821 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/mcp3021.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6683.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6775-core.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6775.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct7802.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pc87360.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pc87427.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pcf8591.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-19T02:33:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/powr1220.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch5627.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch5636.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sht15.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sht21.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/shtc1.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sis5595.ko 2024-03-19T02:33:04.823 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47b397.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47m1.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47m192.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/thmc50.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp102.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp103.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp401.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp421.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/via-cputemp.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/via686a.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/vt1211.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/vt8231.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83627ehf.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83627hf.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83781d.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83791d.ko 2024-03-19T02:33:04.824 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83792d.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83793.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83795.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83l785ts.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83l786ng.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-19T02:33:04.825 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-19T02:33:04.826 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-dev.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-mux.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-smbus.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-stub.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_core.ko 2024-03-19T02:33:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/cxgb4 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx4 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx5 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mthca 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-19T02:33:04.828 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/ipoib 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/iser 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/isert 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srp 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srpt 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-19T02:33:04.829 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/gameport.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/joydev.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/keyboard 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/appletouch.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/bcm5974.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/cyapatp.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/sermouse.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-19T02:33:04.830 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/rmi4 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/altera_ps2.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/arc_ps2.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/serio_raw.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/sparse-keymap.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-blinkm.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-lm3530.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-lp3944.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-ss4200.ko 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger 2024-03-19T02:33:04.831 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/bcache 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/bcache/bcache.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-bio-prison.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-cache-smq.ko 2024-03-19T02:33:04.832 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-cache.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-crypt.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-delay.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-flakey.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-log-userspace.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-multipath.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-queue-length.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-raid.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-round-robin.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-service-time.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-switch.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-thin-pool.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-verity.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/persistent-data 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-19T02:33:04.833 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid0.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid1.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid10.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid456.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptbase.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptctl.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptfc.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptlan.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptsas.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptscsih.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptspi.ko 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd 2024-03-19T02:33:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/lpc_ich.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/lpc_sch.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/mfd-core.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/sm501.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/viperboard.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/vx855.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/wl1273-core.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/altera-stapl 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/apds9802als.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/apds990x.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/bh1770glc.ko 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/cb710 2024-03-19T02:33:04.835 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/cb710/cb710.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/at24.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/max6875.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/enclosure.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/hpilo.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/isl29003.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/isl29020.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei-me.ko 2024-03-19T02:33:04.836 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei-txe.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-gru 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tifm_7xx1.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tifm_core.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tsl2550.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_balloon.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_vmci 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-19T02:33:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/bonding 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/bonding/bonding.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/b53 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6xxx 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dummy.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/eql.ko 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com 2024-03-19T02:33:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/adaptec 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/agere 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/alteon 2024-03-19T02:33:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/altera 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/alx 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-19T02:33:04.840 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/genet 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade/bna 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-19T02:33:04.841 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco/enic 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-19T02:33:04.842 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dnet.ko 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex/benet 2024-03-19T02:33:04.843 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ethoc.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/fealnx.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000e 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/fm10k 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/i40e 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/iavf 2024-03-19T02:33:04.844 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igb 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igbvf 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/jme.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-19T02:33:04.845 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/micrel 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-19T02:33:04.846 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/neterion 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/nvidia 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-19T02:33:04.847 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rdc 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rocker 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sfc 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/silan 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis 2024-03-19T02:33:04.848 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-19T02:33:04.849 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/tehuti 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ti 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/xircom 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/geneve.ko 2024-03-19T02:33:04.850 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/hyperv 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ieee802154 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ifb.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ipvlan 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/macvlan.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/macvtap.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-19T02:33:04.851 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mii.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/net_failover.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/netconsole.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/nlmon.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/amd.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm87xx.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/broadcom.ko 2024-03-19T02:33:04.852 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/cicada.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/davicom.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/dp83640.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/et1011c.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/icplus.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/lxt.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/marvell.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/micrel.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/national.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/phylink.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/qsemi.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/realtek.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/smsc.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/ste10Xp.ko 2024-03-19T02:33:04.853 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/vitesse.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_async.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pppoe.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pppox.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pptp.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip/slhc.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip/slip.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/sungem_phy.ko 2024-03-19T02:33:04.854 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/tap.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_random.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/tun.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/veth.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/virtio_net.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vmxnet3 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vxlan 2024-03-19T02:33:04.855 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vxlan/vxlan.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netback 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netfront.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ntb 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ntb/ntb.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host/nvme-core.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host/nvme.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_cs.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_pc.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_serial.ko 2024-03-19T02:33:04.856 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/hotplug 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/pcie 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/xen-pcifront.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/i82092.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/pd6729.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl/intel 2024-03-19T02:33:04.857 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/acerhdf.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/hdaps.ko 2024-03-19T02:33:04.858 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel_ips.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/wmi.ko 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap 2024-03-19T02:33:04.859 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps_parport.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/pps_core.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp/ptp.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp/ptp_kvm.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-19T02:33:04.860 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-em3027.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-19T02:33:04.861 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-max6900.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-x1205.ko 2024-03-19T02:33:04.862 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-9xxx.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-sas.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-xxxx.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/BusLogic.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/a100u2w.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aacraid 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/advansys.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/am53c974.ko 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/arcmsr 2024-03-19T02:33:04.863 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/atp870u.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/be2iscsi 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bfa 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bfa/bfa.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2fc 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2i 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-19T02:33:04.864 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ch.ko 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/csiostor 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-19T02:33:04.870 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/dc395x.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/dmx3191d.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esas2r 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esp_scsi.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-19T02:33:04.871 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fnic 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fnic/fnic.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hpsa.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hptiop.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hv_storvsc.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/initio.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ipr.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ips.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/isci 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/isci/isci.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libfc 2024-03-19T02:33:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libfc/libfc.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libiscsi.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libsas 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libsas/libsas.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/lpfc 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mpt3sas 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvsas 2024-03-19T02:33:04.873 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvumi.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pm8001 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pmcraid.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla1280.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla4xxx 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/raid_class.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_debug.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-19T02:33:04.874 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ses.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/st.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/stex.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/sym53c8xx_2 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/virtio_scsi.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/wd719x.ko 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ssb 2024-03-19T02:33:04.875 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ssb/ssb.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/iscsi 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/loopback 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_file.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_iblock.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_mod.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_pscsi.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_user.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/tcm_fc 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal 2024-03-19T02:33:04.876 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-19T02:33:04.877 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/ipwireless 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/n_gsm.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/n_hdlc.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/nozomi.ko 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/8250 2024-03-19T02:33:04.878 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/arc_uart.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/jsm 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/core 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/host 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_aec.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_cif.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_pci_generic.ko 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_sercos3.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/cdc-acm.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/usblp.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/usbtmc.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/host 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image/mdc800.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image/microtek.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/adutux.ko 2024-03-19T02:33:04.880 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/appledisplay.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/emi26.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/emi62.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/ezusb.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/idmouse.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/iowarrior.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/ldusb.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/legousbtower.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/sisusbvga 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usb3503.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usblcd.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-19T02:33:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/uss720.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/yurex.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/phy 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/phy/phy-generic.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/aircable.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ark3116.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ch341.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cp210x.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cyberjack.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-19T02:33:04.882 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/empeg.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/io_ti.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ipaq.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ipw.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ir-usb.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/keyspan.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mct_u232.ko 2024-03-19T02:33:04.883 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mos7720.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mos7840.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/navman.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/omninet.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/opticon.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/option.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/oti6858.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/pl2303.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/qcaux.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/qcserial.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/quatech2.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/safe_serial.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/sierra.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-19T02:33:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ssu100.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/symbolserial.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb_debug.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/visor.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/whiteheat.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/uas.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-19T02:33:04.885 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-karma.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/usb-storage.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio 2024-03-19T02:33:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/vfio.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_net.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_scsi.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/apple_bl.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/lcd.ko 2024-03-19T02:33:04.887 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_balloon.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_mmio.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_ring.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters/ds2482.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters/ds2490.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-19T02:33:04.888 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/wire.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/hpwdt.ko 2024-03-19T02:33:04.889 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/i6300esb.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ib700wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ibmasr.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/it87_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/machzwd.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/nv_tco.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/softdog.ko 2024-03-19T02:33:04.890 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/via_wdt.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/wdt_pci.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/xen_wdt.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-evtchn.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-gntalloc.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-gntdev.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-pciback 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-19T02:33:04.891 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-privcmd.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xenfs 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/9p 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/9p/9p.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/affs 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/affs/affs.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/befs 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/befs/befs.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/binfmt_misc.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/btrfs 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/btrfs/btrfs.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cachefiles 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cachefiles/cachefiles.ko 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ceph 2024-03-19T02:33:04.892 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ceph/ceph.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/coda 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/coda/coda.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cramfs 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cramfs/cramfs.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/dlm 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/dlm/dlm.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ecryptfs 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/f2fs 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/f2fs/f2fs.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/fat.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/msdos.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/vfat.ko 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse 2024-03-19T02:33:04.893 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse/cuse.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse/fuse.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/gfs2 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/gfs2/gfs2.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfs 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfs/hfs.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfsplus 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfsplus/hfsplus.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/isofs 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/isofs/isofs.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/jfs 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/jfs/jfs.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/lockd 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/lockd/lockd.ko 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/minix 2024-03-19T02:33:04.894 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/minix/minix.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/netfs 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/netfs/netfs.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/blocklayout 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/filelayout 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/flexfilelayout 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfs.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfsv3.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfsv4.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common/grace.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common/nfs_acl.ko 2024-03-19T02:33:04.895 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfsd 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfsd/nfsd.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nilfs2 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nilfs2/nilfs2.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-celtic.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-centeuro.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-croatian.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-cyrillic.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-gaelic.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-greek.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-iceland.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-inuit.ko 2024-03-19T02:33:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-roman.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-romanian.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-turkish.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1250.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1251.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1255.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp737.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp775.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp850.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp852.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp855.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp857.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp860.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp861.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp862.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp863.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp864.ko 2024-03-19T02:33:04.897 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp865.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp866.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp869.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp874.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp932.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp936.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp949.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp950.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_euc-jp.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-1.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-13.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-14.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-15.ko 2024-03-19T02:33:04.898 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-2.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-3.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-4.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-5.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-6.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-7.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-9.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-r.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-ru.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-u.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_utf8.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/cluster 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlm 2024-03-19T02:33:04.899 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlmfs 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/overlayfs 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/overlayfs/overlay.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/pstore 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/pstore/ramoops.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/reiserfs 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/reiserfs/reiserfs.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/romfs 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/romfs/romfs.ko 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb 2024-03-19T02:33:04.900 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/client 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/client/cifs.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common/cifs_arc4.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common/cifs_md4.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/squashfs 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/squashfs/squashfs.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/sysv 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/sysv/sysv.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/udf 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/udf/udf.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ufs 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ufs/ufs.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/xfs 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/xfs/xfs.ko 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel 2024-03-19T02:33:04.901 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/rcu 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/rcu/rcutorture.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/torture.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/trace 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/asn1_encoder.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crc-itu-t.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crc8.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto/libarc4.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto/libdes.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/libcrc32c.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lru_cache.ko 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4 2024-03-19T02:33:04.902 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4/lz4_compress.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4/lz4hc_compress.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/math 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/math/cordic.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/raid6 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/raid6/raid6_pq.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/reed_solomon 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_bm.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_fsm.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_kmp.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/zstd 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/zstd/zstd_compress.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/mm 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/mm/hwpoison-inject.ko 2024-03-19T02:33:04.903 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/6lowpan.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_dest.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_fragment.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_hop.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_mobility.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_routing.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_udp.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/garp.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/mrp.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/p8022.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/psnap.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/stp.ko 2024-03-19T02:33:04.904 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/8021q 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/8021q/8021q.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_fd.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_rdma.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_virtio.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/batman-adv 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/batman-adv/batman-adv.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/br_netfilter.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/bridge.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-19T02:33:04.905 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-19T02:33:04.906 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtables.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ceph 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ceph/libceph.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core/failover.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core/pktgen.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_diag.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_ipv4.ko 2024-03-19T02:33:04.907 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_ipv6.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dns_resolver 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dns_resolver/dns_resolver.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/dsa_core.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_brcm.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_dsa.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_none.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_trailer.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/6lowpan 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/ieee802154.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-19T02:33:04.908 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ah4.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/esp4.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/fou.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/gre.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/inet_diag.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_gre.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_tunnel.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_vti.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ipcomp.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ipip.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-19T02:33:04.909 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-19T02:33:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_bic.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_dctcp.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_diag.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_highspeed.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_htcp.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_hybla.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_illinois.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_lp.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_scalable.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_vegas.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_veno.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_westwood.ko 2024-03-19T02:33:04.911 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_yeah.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tunnel4.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/udp_diag.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/udp_tunnel.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ah6.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/esp6.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/fou6.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_tunnel.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_vti.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ipcomp6.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-19T02:33:04.912 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-19T02:33:04.913 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/sit.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/tunnel6.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/key 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/key/af_key.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_core.ko 2024-03-19T02:33:04.914 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_eth.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ip.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ip6.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_netlink.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ppp.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/llc 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/llc/llc.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mac802154 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mac802154/mac802154.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mpls 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mpls/mpls_gso.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-19T02:33:04.915 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-19T02:33:04.916 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conncount.ko 2024-03-19T02:33:04.917 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-19T02:33:04.918 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_flow_table.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_log_syslog.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_irc.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_sip.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_tables.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_log.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-19T02:33:04.919 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_chain_nat.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_compat.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_ct.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_fib.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_hash.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_limit.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_log.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_masq.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_nat.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_numgen.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_queue.ko 2024-03-19T02:33:04.920 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_quota.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_redir.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_reject.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_reject_inet.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_socket.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_tunnel.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_AUDIT.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CT.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_DSCP.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_HL.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_HMARK.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_LED.ko 2024-03-19T02:33:04.921 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_LOG.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NETMAP.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NFLOG.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_RATEEST.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_SECMARK.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TEE.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TPROXY.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TRACE.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_addrtype.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_bpf.ko 2024-03-19T02:33:04.922 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cgroup.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cluster.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_comment.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connbytes.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connlabel.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connlimit.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connmark.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_conntrack.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cpu.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_dccp.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_devgroup.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_dscp.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ecn.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_esp.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_hashlimit.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_helper.ko 2024-03-19T02:33:04.923 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_hl.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ipcomp.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_iprange.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ipvs.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_l2tp.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_length.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_limit.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_mac.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_mark.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_multiport.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_nat.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_nfacct.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_osf.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_owner.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_physdev.ko 2024-03-19T02:33:04.924 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_pkttype.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_policy.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_quota.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_rateest.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_realm.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_recent.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_sctp.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_set.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_socket.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_state.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_statistic.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_string.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_tcpmss.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_time.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_u32.ko 2024-03-19T02:33:04.925 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netlink 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netlink/netlink_diag.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/nsh 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/nsh/nsh.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/openvswitch.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-geneve.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-gre.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-vxlan.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/packet 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/packet/af_packet_diag.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds_rdma.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds_tcp.ko 2024-03-19T02:33:04.926 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_csum.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_gact.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_mirred.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_nat.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_pedit.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_police.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_simple.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_skbedit.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_vlan.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_basic.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_bpf.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_flow.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_fw.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_route.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_u32.ko 2024-03-19T02:33:04.927 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_cmp.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_ipset.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_meta.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_nbyte.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_text.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_u32.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_choke.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_codel.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_drr.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_fq.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_gred.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_hfsc.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_hhf.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_htb.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_ingress.ko 2024-03-19T02:33:04.928 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_mqprio.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_mqprio_lib.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_multiq.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_netem.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_pie.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_plug.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_prio.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_qfq.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_red.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_sfb.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_sfq.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_tbf.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_teql.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp/sctp.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp/sctp_diag.ko 2024-03-19T02:33:04.929 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/sunrpc.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/xprtrdma 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc/diag.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc/tipc.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/unix 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/unix/unix_diag.ko 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock 2024-03-19T02:33:04.930 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/xfrm 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/encrypted-keys 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/trusted-keys 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/trusted-keys/trusted.ko 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt 2024-03-19T02:33:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt/lib 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt/lib/irqbypass.ko 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.alias 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.alias.bin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.alias.bin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.bin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.modinfo 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.dep 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.dep.bin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.devname 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.order 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.softdep 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.symbols 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.symbols.bin 2024-03-19T02:33:04.932 INFO:teuthology.orchestra.run.smithi092.stdout:/lib/modules/6.8.0-g908b6b276e8f/vmlinuz 2024-03-19T02:33:04.936 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g908b6b276e8f 2024-03-19T02:33:04.936 DEBUG:teuthology.orchestra.run.smithi092:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:postinstall scriptlet (using /bin/sh): 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout: /usr/bin/kernel-install add 6.8.0-g908b6b276e8f /lib/modules/6.8.0-g908b6b276e8f/vmlinuz 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:fi 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:for file in vmlinuz System.map config; do 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout: if ! cmp --silent "/lib/modules/6.8.0-g908b6b276e8f/${file}" "/boot/${file}-6.8.0-g908b6b276e8f"; then 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout: cp "/lib/modules/6.8.0-g908b6b276e8f/${file}" "/boot/${file}-6.8.0-g908b6b276e8f" 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout: fi 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:done 2024-03-19T02:33:04.959 INFO:teuthology.orchestra.run.smithi092.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:new-kernel-pkg --remove 6.8.0-g908b6b276e8f --rminitrd --initrdfile=/boot/initramfs-6.8.0-g908b6b276e8f.img 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-install remove 6.8.0-g908b6b276e8f 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:fi 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/update-bootloader --remove 6.8.0-g908b6b276e8f 2024-03-19T02:33:04.960 INFO:teuthology.orchestra.run.smithi092.stdout:fi 2024-03-19T02:33:04.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-03-19T02:33:05.037 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:33:05.038 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:33:05.039 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:33:05.039 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-03-19T02:33:05.039 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:33:05.039 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:33:05.040 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:33:05.040 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 6.8.0-g908b6b276e8f 2024-03-19T02:33:05.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:33:05.721 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-03-19T02:33:07.168 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:33:07.207 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-03-19T02:33:07.210 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-19T02:33:07.265 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.YC8jf5oAza 2024-03-19T02:33:07.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YC8jf5oAza 2024-03-19T02:33:07.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.YC8jf5oAza 2024-03-19T02:33:07.477 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.YC8jf5oAza is 6KB 2024-03-19T02:33:07.497 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.YC8jf5oAza 2024-03-19T02:33:07.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:33:07.575 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:33:07.575 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:33:07.575 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:33:07.575 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g908b6b276e8f.conf 2024-03-19T02:33:07.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g908b6b276e8f 2024-03-19T02:33:07.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-03-19T02:33:37.708 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-19T02:33:37.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:33:37.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:33:56.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-19T02:34:05.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:34:05.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:34:08.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-19T02:34:20.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:34:20.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:35:20.322 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T02:35:35.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:35:35.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:35:35.677 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-19T02:35:36.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:35:36.354 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g908b6b276e8f"... 2024-03-19T02:35:36.354 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-19T02:35:36.402 INFO:teuthology.orchestra.run.smithi092.stdout:6.8.0-g908b6b276e8f 2024-03-19T02:35:36.403 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g908b6b276e8f vs 6.8.0-g908b6b276e8f 2024-03-19T02:35:36.403 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:35:36.403 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:35:37.406 DEBUG:teuthology.parallel:result is None 2024-03-19T02:35:37.407 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T02:35:37.415 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T02:35:37.415 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:35:37.422 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:35:37.471 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T02:35:37.478 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T02:35:37.502 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T02:35:37.502 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:35:37.508 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:35:37.619 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T02:35:37.626 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T02:35:37.626 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:35:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:35:37.643 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:35:37.650 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:35:37.747 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:35:37.766 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:35:37.789 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:35:37.855 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:35:37.860 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T02:35:37.870 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T02:35:37.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:35:37.876 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:35:37.960 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T02:35:37.970 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T02:35:37.970 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:35:37.975 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:35:38.024 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:35:38.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:35:38.221 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:35:38.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:35:38.403 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:35:38.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:35:38.499 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:35:38.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:35:38.595 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:35:38.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:35:38.723 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:35:38.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:35:38.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-03-19T02:35:38.800 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-03-19T02:35:38.872 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:35:38.925 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:35:39.260 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T02:35:39.271 INFO:teuthology.task.internal:Starting timer... 2024-03-19T02:35:39.271 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T02:35:39.301 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T02:35:39.329 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-19T02:35:39.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-03-19T02:35:39.452 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-03-19T02:35:39.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-03-19T02:35:39.520 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-03-19T02:35:39.521 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T02:35:39.522 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-03-19T02:35:39.584 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-03-19T02:35:39.584 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-03-19T02:35:39.616 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-03-19T02:35:39.616 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-03-19T02:35:39.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:35:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:35:39.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:35:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:35:39.732 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T02:35:39.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:35:39.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:35:39.851 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T02:35:39.863 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-19T02:35:39.869 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-19T02:35:39.995 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T02:35:40.012 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T02:35:40.013 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T02:44:57.407 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-03-19T02:44:57.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:44:57.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:44:57.568 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-19T02:44:57.722 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-19T02:44:57.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-19T02:44:57.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:44:57.813 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-19T02:44:57.882 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-19T02:44:57.882 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T02:44:57.895 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T02:44:57.895 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:44:57.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:44:57.902 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:44:57.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:44:57.956 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:44:57.973 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:44:57.994 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:44:58.004 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-03-19T02:44:58.018 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-03-19T02:44:58.034 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:44:58.046 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:44:58.051 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:44:58.095 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:44:58.099 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:44:58.099 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-19T02:44:58.099 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.100 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.100 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.100 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-03-19T02:44:58.182 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-03-19T02:44:58.231 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:44:58.282 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:44:58.407 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:44:58.415 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:44:58.416 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-19T02:44:58.416 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.416 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.416 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.416 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:44:58.417 INFO:teuthology.run_tasks:Running task install... 2024-03-19T02:44:58.426 DEBUG:teuthology.task.install:project ceph 2024-03-19T02:44:58.426 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T02:44:58.426 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T02:44:58.426 INFO:teuthology.task.install:Using flavor: default 2024-03-19T02:44:58.432 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T02:44:58.432 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T02:44:58.433 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:44:58.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:44:58.436 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:44:58.436 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:44:58.673 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:44:58.673 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:44:58.678 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:44:58.678 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:44:58.782 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:44:58.782 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:44:58.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:44:58.812 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:44:58.812 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:44:58.855 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:44:58.855 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:44:58.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:44:58.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:44:58.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:44:58.953 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:44:58.961 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:44:59.027 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-03-19T02:44:59.029 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-03-19T02:44:59.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:44:59.222 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:44:59.264 INFO:teuthology.orchestra.run.smithi159.stdout:74 files removed 2024-03-19T02:44:59.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:44:59.312 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-03-19T02:44:59.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-03-19T02:44:59.678 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-03-19T02:44:59.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:45:00.049 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-03-19T02:45:00.391 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-19T02:45:00.684 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-19T02:45:00.808 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-19T02:45:01.235 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-19T02:45:01.252 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-19T02:45:01.727 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-19T02:45:02.380 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-19T02:45:02.678 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-19T02:45:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-19T02:45:06.160 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-19T02:45:11.569 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-19T02:45:18.044 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:02 2024-03-19T02:45:20.362 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-19T02:45:22.632 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-19T02:45:27.679 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-19T02:45:30.018 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:45:30.019 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:45:30.033 INFO:teuthology.orchestra.run.smithi159.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:45:30.034 INFO:teuthology.orchestra.run.smithi159.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:45:30.035 INFO:teuthology.orchestra.run.smithi159.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:45:30.037 INFO:teuthology.orchestra.run.smithi159.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:45:30.039 INFO:teuthology.orchestra.run.smithi159.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:45:30.042 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:45:30.043 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:45:30.124 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:45:30.151 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:45:30.152 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:45:30.153 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:45:30.154 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:45:30.155 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:45:30.156 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:45:30.157 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:45:30.158 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:45:30.159 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:45:30.160 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:45:30.161 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:45:30.162 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:45:30.163 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:Install 150 Packages 2024-03-19T02:45:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 24 Packages 2024-03-19T02:45:30.165 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:45:30.165 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 224 M 2024-03-19T02:45:30.165 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-19T02:45:30.589 INFO:teuthology.orchestra.run.smithi159.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-19T02:45:30.923 INFO:teuthology.orchestra.run.smithi159.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 2.6 MB/s | 891 kB 00:00 2024-03-19T02:45:31.023 INFO:teuthology.orchestra.run.smithi159.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 1.4 MB/s | 145 kB 00:00 2024-03-19T02:45:31.148 INFO:teuthology.orchestra.run.smithi159.stdout:(4/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 6.9 MB/s | 5.4 MB 00:00 2024-03-19T02:45:31.299 INFO:teuthology.orchestra.run.smithi159.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 10 MB/s | 1.5 MB 00:00 2024-03-19T02:45:31.517 INFO:teuthology.orchestra.run.smithi159.stdout:(6/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 4.8 MB/s | 2.4 MB 00:00 2024-03-19T02:45:31.718 INFO:teuthology.orchestra.run.smithi159.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-19T02:45:32.501 INFO:teuthology.orchestra.run.smithi159.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 13 MB/s | 9.8 MB 00:00 2024-03-19T02:45:32.593 INFO:teuthology.orchestra.run.smithi159.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 275 kB/s | 25 kB 00:00 2024-03-19T02:45:33.201 INFO:teuthology.orchestra.run.smithi159.stdout:(10/174): ceph-common-19.0.0-2206.gb6ae2491.el9 7.3 MB/s | 21 MB 00:02 2024-03-19T02:45:33.285 INFO:teuthology.orchestra.run.smithi159.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 2.6 MB/s | 221 kB 00:00 2024-03-19T02:45:33.411 INFO:teuthology.orchestra.run.smithi159.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 216 kB/s | 27 kB 00:00 2024-03-19T02:45:33.494 INFO:teuthology.orchestra.run.smithi159.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 381 kB/s | 32 kB 00:00 2024-03-19T02:45:33.678 INFO:teuthology.orchestra.run.smithi159.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 4.0 MB/s | 746 kB 00:00 2024-03-19T02:45:33.787 INFO:teuthology.orchestra.run.smithi159.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.5 MB/s | 163 kB 00:00 2024-03-19T02:45:33.895 INFO:teuthology.orchestra.run.smithi159.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.1 MB/s | 127 kB 00:00 2024-03-19T02:45:34.020 INFO:teuthology.orchestra.run.smithi159.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 3.9 MB/s | 501 kB 00:00 2024-03-19T02:45:34.371 INFO:teuthology.orchestra.run.smithi159.stdout:(18/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 5.9 MB/s | 17 MB 00:02 2024-03-19T02:45:34.462 INFO:teuthology.orchestra.run.smithi159.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 494 kB/s | 45 kB 00:00 2024-03-19T02:45:34.563 INFO:teuthology.orchestra.run.smithi159.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.3 MB/s | 131 kB 00:00 2024-03-19T02:45:34.696 INFO:teuthology.orchestra.run.smithi159.stdout:(21/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 7.6 MB/s | 5.1 MB 00:00 2024-03-19T02:45:34.721 INFO:teuthology.orchestra.run.smithi159.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.0 MB/s | 162 kB 00:00 2024-03-19T02:45:34.797 INFO:teuthology.orchestra.run.smithi159.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.1 MB/s | 321 kB 00:00 2024-03-19T02:45:34.831 INFO:teuthology.orchestra.run.smithi159.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.7 MB/s | 298 kB 00:00 2024-03-19T02:45:34.965 INFO:teuthology.orchestra.run.smithi159.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 595 kB/s | 99 kB 00:00 2024-03-19T02:45:34.990 INFO:teuthology.orchestra.run.smithi159.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 537 kB/s | 85 kB 00:00 2024-03-19T02:45:35.081 INFO:teuthology.orchestra.run.smithi159.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-19T02:45:35.191 INFO:teuthology.orchestra.run.smithi159.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 313 kB/s | 34 kB 00:00 2024-03-19T02:45:35.291 INFO:teuthology.orchestra.run.smithi159.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.4 MB/s | 139 kB 00:00 2024-03-19T02:45:35.382 INFO:teuthology.orchestra.run.smithi159.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 7.4 MB/s | 3.1 MB 00:00 2024-03-19T02:45:35.967 INFO:teuthology.orchestra.run.smithi159.stdout:(31/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 14 MB/s | 48 MB 00:03 2024-03-19T02:45:36.008 INFO:teuthology.orchestra.run.smithi159.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 2.5 MB/s | 1.8 MB 00:00 2024-03-19T02:45:36.050 INFO:teuthology.orchestra.run.smithi159.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.9 MB/s | 248 kB 00:00 2024-03-19T02:45:36.075 INFO:teuthology.orchestra.run.smithi159.stdout:(34/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 740 kB/s | 49 kB 00:00 2024-03-19T02:45:36.126 INFO:teuthology.orchestra.run.smithi159.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 219 kB/s | 16 kB 00:00 2024-03-19T02:45:36.259 INFO:teuthology.orchestra.run.smithi159.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-03-19T02:45:36.284 INFO:teuthology.orchestra.run.smithi159.stdout:(37/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 1.2 MB/s | 264 kB 00:00 2024-03-19T02:45:36.318 INFO:teuthology.orchestra.run.smithi159.stdout:(38/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 3.8 MB/s | 749 kB 00:00 2024-03-19T02:45:36.368 INFO:teuthology.orchestra.run.smithi159.stdout:(39/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 168 kB/s | 18 kB 00:00 2024-03-19T02:45:36.619 INFO:teuthology.orchestra.run.smithi159.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 471 kB/s | 106 kB 00:00 2024-03-19T02:45:36.702 INFO:teuthology.orchestra.run.smithi159.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-19T02:45:36.753 INFO:teuthology.orchestra.run.smithi159.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 920 kB 00:00 2024-03-19T02:45:36.778 INFO:teuthology.orchestra.run.smithi159.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 1.9 MB/s | 307 kB 00:00 2024-03-19T02:45:36.794 INFO:teuthology.orchestra.run.smithi159.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 346 kB/s | 32 kB 00:00 2024-03-19T02:45:36.970 INFO:teuthology.orchestra.run.smithi159.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 57 kB/s | 10 kB 00:00 2024-03-19T02:45:36.987 INFO:teuthology.orchestra.run.smithi159.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 89 kB/s | 17 kB 00:00 2024-03-19T02:45:37.045 INFO:teuthology.orchestra.run.smithi159.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-03-19T02:45:37.071 INFO:teuthology.orchestra.run.smithi159.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-19T02:45:37.096 INFO:teuthology.orchestra.run.smithi159.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 407 kB/s | 44 kB 00:00 2024-03-19T02:45:37.121 INFO:teuthology.orchestra.run.smithi159.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-19T02:45:37.147 INFO:teuthology.orchestra.run.smithi159.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 46 kB 00:00 2024-03-19T02:45:37.180 INFO:teuthology.orchestra.run.smithi159.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 578 kB 00:00 2024-03-19T02:45:37.205 INFO:teuthology.orchestra.run.smithi159.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-19T02:45:37.231 INFO:teuthology.orchestra.run.smithi159.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-19T02:45:37.256 INFO:teuthology.orchestra.run.smithi159.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-19T02:45:37.281 INFO:teuthology.orchestra.run.smithi159.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-03-19T02:45:37.440 INFO:teuthology.orchestra.run.smithi159.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-19T02:45:37.482 INFO:teuthology.orchestra.run.smithi159.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 6.7 MB/s | 1.5 MB 00:00 2024-03-19T02:45:37.515 INFO:teuthology.orchestra.run.smithi159.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.4 MB/s | 811 kB 00:00 2024-03-19T02:45:37.540 INFO:teuthology.orchestra.run.smithi159.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-19T02:45:37.741 INFO:teuthology.orchestra.run.smithi159.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-19T02:45:37.766 INFO:teuthology.orchestra.run.smithi159.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 349 kB/s | 78 kB 00:00 2024-03-19T02:45:37.799 INFO:teuthology.orchestra.run.smithi159.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-03-19T02:45:37.933 INFO:teuthology.orchestra.run.smithi159.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-19T02:45:37.958 INFO:teuthology.orchestra.run.smithi159.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-19T02:45:37.983 INFO:teuthology.orchestra.run.smithi159.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 100 kB/s | 18 kB 00:00 2024-03-19T02:45:38.009 INFO:teuthology.orchestra.run.smithi159.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-19T02:45:38.034 INFO:teuthology.orchestra.run.smithi159.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-19T02:45:38.159 INFO:teuthology.orchestra.run.smithi159.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-19T02:45:38.184 INFO:teuthology.orchestra.run.smithi159.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-19T02:45:38.210 INFO:teuthology.orchestra.run.smithi159.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-19T02:45:38.235 INFO:teuthology.orchestra.run.smithi159.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-03-19T02:45:38.260 INFO:teuthology.orchestra.run.smithi159.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-19T02:45:38.285 INFO:teuthology.orchestra.run.smithi159.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-19T02:45:38.311 INFO:teuthology.orchestra.run.smithi159.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-19T02:45:38.336 INFO:teuthology.orchestra.run.smithi159.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-19T02:45:38.361 INFO:teuthology.orchestra.run.smithi159.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-19T02:45:38.378 INFO:teuthology.orchestra.run.smithi159.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 640 kB/s | 43 kB 00:00 2024-03-19T02:45:38.787 INFO:teuthology.orchestra.run.smithi159.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-19T02:45:38.812 INFO:teuthology.orchestra.run.smithi159.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 95 kB/s | 41 kB 00:00 2024-03-19T02:45:38.837 INFO:teuthology.orchestra.run.smithi159.stdout:(81/174): python3-toml-0.10.2-6.el9.noarch.rpm 85 kB/s | 39 kB 00:00 2024-03-19T02:45:38.863 INFO:teuthology.orchestra.run.smithi159.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-19T02:45:38.888 INFO:teuthology.orchestra.run.smithi159.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 754 kB/s | 56 kB 00:00 2024-03-19T02:45:38.913 INFO:teuthology.orchestra.run.smithi159.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-19T02:45:38.938 INFO:teuthology.orchestra.run.smithi159.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 834 kB/s | 63 kB 00:00 2024-03-19T02:45:38.964 INFO:teuthology.orchestra.run.smithi159.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-19T02:45:38.989 INFO:teuthology.orchestra.run.smithi159.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-19T02:45:39.014 INFO:teuthology.orchestra.run.smithi159.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-19T02:45:39.048 INFO:teuthology.orchestra.run.smithi159.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-19T02:45:39.073 INFO:teuthology.orchestra.run.smithi159.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-19T02:45:39.098 INFO:teuthology.orchestra.run.smithi159.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-19T02:45:39.623 INFO:teuthology.orchestra.run.smithi159.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 326 kB/s | 187 kB 00:00 2024-03-19T02:45:39.649 INFO:teuthology.orchestra.run.smithi159.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 355 kB/s | 195 kB 00:00 2024-03-19T02:45:39.732 INFO:teuthology.orchestra.run.smithi159.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-19T02:45:39.757 INFO:teuthology.orchestra.run.smithi159.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-19T02:45:39.808 INFO:teuthology.orchestra.run.smithi159.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-19T02:45:39.825 INFO:teuthology.orchestra.run.smithi159.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-19T02:45:39.942 INFO:teuthology.orchestra.run.smithi159.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-19T02:45:39.967 INFO:teuthology.orchestra.run.smithi159.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-19T02:45:40.026 INFO:teuthology.orchestra.run.smithi159.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 598 kB/s | 35 kB 00:00 2024-03-19T02:45:40.051 INFO:teuthology.orchestra.run.smithi159.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.9 MB/s | 216 kB 00:00 2024-03-19T02:45:40.084 INFO:teuthology.orchestra.run.smithi159.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-19T02:45:40.110 INFO:teuthology.orchestra.run.smithi159.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-19T02:45:40.137 INFO:teuthology.orchestra.run.smithi159.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 424 kB/s | 22 kB 00:00 2024-03-19T02:45:40.304 INFO:teuthology.orchestra.run.smithi159.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.3 MB/s | 863 kB 00:00 2024-03-19T02:45:40.604 INFO:teuthology.orchestra.run.smithi159.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-19T02:45:40.671 INFO:teuthology.orchestra.run.smithi159.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 293 kB/s | 19 kB 00:00 2024-03-19T02:45:40.696 INFO:teuthology.orchestra.run.smithi159.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 791 kB/s | 309 kB 00:00 2024-03-19T02:45:40.755 INFO:teuthology.orchestra.run.smithi159.stdout:(109/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.1 MB/s | 4.1 MB 00:00 2024-03-19T02:45:40.780 INFO:teuthology.orchestra.run.smithi159.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 310 kB/s | 26 kB 00:00 2024-03-19T02:45:40.855 INFO:teuthology.orchestra.run.smithi159.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 905 kB/s | 67 kB 00:00 2024-03-19T02:45:40.973 INFO:teuthology.orchestra.run.smithi159.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-19T02:45:40.998 INFO:teuthology.orchestra.run.smithi159.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-19T02:45:41.073 INFO:teuthology.orchestra.run.smithi159.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-19T02:45:41.098 INFO:teuthology.orchestra.run.smithi159.stdout:(115/174): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-19T02:45:41.140 INFO:teuthology.orchestra.run.smithi159.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-19T02:45:41.165 INFO:teuthology.orchestra.run.smithi159.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 471 kB/s | 43 kB 00:00 2024-03-19T02:45:41.191 INFO:teuthology.orchestra.run.smithi159.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 349 kB/s | 32 kB 00:00 2024-03-19T02:45:41.216 INFO:teuthology.orchestra.run.smithi159.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-19T02:45:41.241 INFO:teuthology.orchestra.run.smithi159.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-19T02:45:41.308 INFO:teuthology.orchestra.run.smithi159.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-19T02:45:41.358 INFO:teuthology.orchestra.run.smithi159.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-03-19T02:45:41.392 INFO:teuthology.orchestra.run.smithi159.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-19T02:45:41.417 INFO:teuthology.orchestra.run.smithi159.stdout:(124/174): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-03-19T02:45:41.501 INFO:teuthology.orchestra.run.smithi159.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-19T02:45:41.526 INFO:teuthology.orchestra.run.smithi159.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-19T02:45:41.551 INFO:teuthology.orchestra.run.smithi159.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 909 kB/s | 144 kB 00:00 2024-03-19T02:45:41.585 INFO:teuthology.orchestra.run.smithi159.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-19T02:45:41.610 INFO:teuthology.orchestra.run.smithi159.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-03-19T02:45:41.636 INFO:teuthology.orchestra.run.smithi159.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 231 kB/s | 19 kB 00:00 2024-03-19T02:45:41.661 INFO:teuthology.orchestra.run.smithi159.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-19T02:45:41.711 INFO:teuthology.orchestra.run.smithi159.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 616 kB/s | 46 kB 00:00 2024-03-19T02:45:41.736 INFO:teuthology.orchestra.run.smithi159.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-19T02:45:41.786 INFO:teuthology.orchestra.run.smithi159.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 771 kB/s | 58 kB 00:00 2024-03-19T02:45:41.821 INFO:teuthology.orchestra.run.smithi159.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-03-19T02:45:41.846 INFO:teuthology.orchestra.run.smithi159.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-19T02:45:41.871 INFO:teuthology.orchestra.run.smithi159.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 195 kB/s | 16 kB 00:00 2024-03-19T02:45:41.896 INFO:teuthology.orchestra.run.smithi159.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-19T02:45:41.921 INFO:teuthology.orchestra.run.smithi159.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-19T02:45:41.947 INFO:teuthology.orchestra.run.smithi159.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-19T02:45:41.972 INFO:teuthology.orchestra.run.smithi159.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi159.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-19T02:45:42.023 INFO:teuthology.orchestra.run.smithi159.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-03-19T02:45:42.048 INFO:teuthology.orchestra.run.smithi159.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-03-19T02:45:42.073 INFO:teuthology.orchestra.run.smithi159.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-19T02:45:42.098 INFO:teuthology.orchestra.run.smithi159.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-19T02:45:42.132 INFO:teuthology.orchestra.run.smithi159.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-19T02:45:42.149 INFO:teuthology.orchestra.run.smithi159.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-19T02:45:42.191 INFO:teuthology.orchestra.run.smithi159.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-19T02:45:42.333 INFO:teuthology.orchestra.run.smithi159.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-19T02:45:42.383 INFO:teuthology.orchestra.run.smithi159.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-19T02:45:42.825 INFO:teuthology.orchestra.run.smithi159.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 5.0 MB/s | 3.4 MB 00:00 2024-03-19T02:45:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 4.1 MB/s | 3.1 MB 00:00 2024-03-19T02:45:43.301 INFO:teuthology.orchestra.run.smithi159.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-03-19T02:45:43.326 INFO:teuthology.orchestra.run.smithi159.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 126 kB/s | 42 kB 00:00 2024-03-19T02:45:43.417 INFO:teuthology.orchestra.run.smithi159.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-19T02:45:43.535 INFO:teuthology.orchestra.run.smithi159.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 342 kB/s | 71 kB 00:00 2024-03-19T02:45:43.560 INFO:teuthology.orchestra.run.smithi159.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 256 kB/s | 36 kB 00:00 2024-03-19T02:45:43.602 INFO:teuthology.orchestra.run.smithi159.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.4 MB/s | 1.8 MB 00:00 2024-03-19T02:45:43.677 INFO:teuthology.orchestra.run.smithi159.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-19T02:45:43.719 INFO:teuthology.orchestra.run.smithi159.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 197 kB/s | 23 kB 00:00 2024-03-19T02:45:43.786 INFO:teuthology.orchestra.run.smithi159.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 998 kB/s | 108 kB 00:00 2024-03-19T02:45:43.836 INFO:teuthology.orchestra.run.smithi159.stdout:(163/174): elfutils-libelf-0.190-2.el9.x86_64.r 638 kB/s | 192 kB 00:00 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi159.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi159.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-19T02:45:43.954 INFO:teuthology.orchestra.run.smithi159.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 543 kB/s | 64 kB 00:00 2024-03-19T02:45:43.979 INFO:teuthology.orchestra.run.smithi159.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 245 kB/s | 29 kB 00:00 2024-03-19T02:45:44.046 INFO:teuthology.orchestra.run.smithi159.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-19T02:45:44.180 INFO:teuthology.orchestra.run.smithi159.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-03-19T02:45:44.263 INFO:teuthology.orchestra.run.smithi159.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.0 MB/s | 2.3 MB 00:00 2024-03-19T02:45:44.305 INFO:teuthology.orchestra.run.smithi159.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 248 kB/s | 31 kB 00:00 2024-03-19T02:45:44.405 INFO:teuthology.orchestra.run.smithi159.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-19T02:45:44.439 INFO:teuthology.orchestra.run.smithi159.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 570 kB/s | 74 kB 00:00 2024-03-19T02:45:44.539 INFO:teuthology.orchestra.run.smithi159.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.7 MB/s | 2.2 MB 00:00 2024-03-19T02:45:44.543 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:45:44.543 INFO:teuthology.orchestra.run.smithi159.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-19T02:45:45.033 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-19T02:45:45.309 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-19T02:45:45.310 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-19T02:45:47.028 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-19T02:45:47.029 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-19T02:45:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-19T02:45:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:45:53.994 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:45:54.112 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:45:54.122 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:45:54.191 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:45:54.245 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:45:54.278 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:45:54.312 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:45:54.355 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:45:54.674 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:45:54.730 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:45:54.851 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:45:54.880 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:45:54.980 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:45:55.009 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:45:55.063 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:45:55.246 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:45:55.308 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:45:55.336 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:45:55.423 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:45:55.445 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:45:55.492 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:45:55.532 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:45:55.580 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:45:55.596 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:45:55.822 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:45:55.852 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:45:55.884 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:45:55.982 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:45:56.070 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:45:56.130 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:45:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:45:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:45:56.480 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:45:56.604 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:45:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:45:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:45:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:45:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:45:56.873 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:45:56.927 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:45:56.965 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:45:57.031 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:45:57.073 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:45:57.117 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:45:57.149 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:45:57.177 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:45:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:45:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:45:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:45:57.354 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:45:57.439 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:45:57.487 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:45:57.570 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:45:57.708 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:45:57.765 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:45:58.314 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:45:58.385 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:45:58.473 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:45:58.521 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:45:58.531 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:45:58.562 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:45:58.634 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:45:58.674 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:45:58.703 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:45:58.948 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:45:59.121 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:45:59.197 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:45:59.241 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:45:59.371 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:45:59.427 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:45:59.676 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:45:59.748 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:45:59.778 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:45:59.815 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:45:59.862 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:45:59.898 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:45:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:45:59.983 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:46:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:46:00.171 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:46:00.202 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:46:00.243 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:46:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:46:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:46:00.417 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:46:00.442 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:46:00.467 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:46:00.539 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:46:00.669 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:46:00.768 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:46:00.810 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:46:00.858 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:46:00.913 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:46:01.100 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:46:01.146 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:46:01.275 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:46:01.311 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:46:01.345 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:46:01.374 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:46:01.402 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:46:01.436 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:46:01.476 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:46:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:46:01.562 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:46:01.638 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:46:01.673 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:46:01.706 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:46:01.747 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:46:01.782 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:46:01.870 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:46:01.961 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:46:02.004 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:46:02.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:46:02.083 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:46:02.180 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:46:02.237 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:46:02.969 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:46:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:46:03.043 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:46:03.091 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:46:03.179 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:46:03.211 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:46:03.387 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:46:03.417 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:46:03.494 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:46:04.136 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:46:04.240 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:46:05.466 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:46:05.559 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:46:05.590 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:46:05.621 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:46:05.649 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:46:05.959 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:46:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:46:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:46:06.108 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:46:06.145 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:46:06.603 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:46:06.776 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:46:07.058 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:46:07.142 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:46:09.442 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:46:09.480 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:46:10.022 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:46:10.059 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:46:10.161 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:46:10.161 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:46:10.161 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:46:10.161 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:10.193 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-03-19T02:46:22.140 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-03-19T02:46:22.141 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-03-19T02:46:22.141 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:22.307 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:46:22.323 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:46:22.324 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:22.324 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:46:22.324 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:23.785 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:46:23.810 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:46:23.811 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:23.811 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:46:23.811 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:46:23.811 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:46:23.811 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:24.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:46:24.059 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:24.495 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:46:24.518 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:24.622 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:46:25.000 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:46:25.030 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:46:25.062 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:46:25.123 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:46:25.153 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:46:25.391 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:46:25.421 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:46:26.394 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:46:26.430 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:46:26.531 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:46:26.692 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:46:26.723 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:46:26.747 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:26.799 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:46:26.810 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:46:26.855 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:46:27.736 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:46:27.759 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:46:27.782 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:46:27.782 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:27.783 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:46:27.783 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:46:27.783 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:46:27.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:27.827 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:46:27.849 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:46:27.849 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:27.849 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:46:27.849 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:27.902 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:46:27.925 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:46:27.925 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:27.925 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:46:27.926 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:46:27.926 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:46:27.926 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:46:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:32.735 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:46:32.773 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:46:32.804 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:46:32.831 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:46:33.008 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:46:33.130 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:46:33.180 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:46:33.219 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:46:33.271 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:46:33.340 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:46:33.372 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:46:33.404 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:46:33.434 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:46:33.488 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:46:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:46:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:46:34.181 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:46:34.222 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:46:34.243 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:46:35.103 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:46:35.184 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:46:35.225 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:46:35.306 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:46:35.338 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:46:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:46:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:46:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:46:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:46:35.503 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:46:35.532 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:46:35.533 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:46:35.618 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:46:35.643 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:46:35.677 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:46:35.702 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:46:35.726 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:46:35.761 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:46:35.799 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:46:35.832 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:46:35.926 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:46:35.926 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:46:35.926 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:46:35.932 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:46:35.939 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:46:35.939 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:46:36.119 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:46:36.119 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:46:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:46:36.251 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:46:36.283 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:46:36.303 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:46:36.336 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:46:36.400 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:46:41.354 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:46:41.355 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:46:41.356 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:46:41.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:46:41.358 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:46:41.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:46:41.360 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:46:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:46:41.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:46:41.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:46:41.364 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:46:41.365 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:46:41.366 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:46:41.367 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:46:44.357 INFO:teuthology.orchestra.run.smithi159.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:46:44.358 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.359 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.360 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:46:44.361 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:46:44.362 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:46:44.363 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:46:44.364 INFO:teuthology.orchestra.run.smithi159.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:46:44.365 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:46:44.366 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:46:44.367 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:46:44.368 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:46:44.369 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-19T02:46:44.819 DEBUG:teuthology.parallel:result is None 2024-03-19T02:49:45.229 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 29 kB/s | 8.0 MB 04:40 2024-03-19T02:49:49.653 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-19T02:49:54.403 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.1 MB 00:00 2024-03-19T02:49:56.154 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-19T02:49:58.139 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-19T02:50:03.973 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-19T02:50:06.511 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:50:06.532 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:50:06.533 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:50:06.534 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:50:06.535 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:50:06.538 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:50:06.541 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:50:06.542 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:50:06.631 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:50:06.652 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:50:06.653 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:50:06.654 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-03-19T02:50:06.655 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:50:06.656 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:50:06.657 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:50:06.658 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:50:06.659 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:50:06.660 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:50:06.661 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:50:06.662 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:50:06.663 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:50:06.664 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:50:06.665 INFO:teuthology.orchestra.run.smithi092.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout:Install 150 Packages 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 24 Packages 2024-03-19T02:50:06.666 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:50:06.667 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 224 M 2024-03-19T02:50:06.667 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-19T02:50:08.057 INFO:teuthology.orchestra.run.smithi092.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-19T02:50:08.257 INFO:teuthology.orchestra.run.smithi092.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 4.4 MB/s | 891 kB 00:00 2024-03-19T02:50:08.324 INFO:teuthology.orchestra.run.smithi092.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.1 MB/s | 145 kB 00:00 2024-03-19T02:50:08.466 INFO:teuthology.orchestra.run.smithi092.stdout:(4/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 8.1 MB/s | 5.4 MB 00:00 2024-03-19T02:50:08.643 INFO:teuthology.orchestra.run.smithi092.stdout:(5/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.4 MB/s | 2.4 MB 00:00 2024-03-19T02:50:08.725 INFO:teuthology.orchestra.run.smithi092.stdout:(6/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-19T02:50:09.026 INFO:teuthology.orchestra.run.smithi092.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-19T02:50:09.897 INFO:teuthology.orchestra.run.smithi092.stdout:(8/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 14 MB/s | 17 MB 00:01 2024-03-19T02:50:09.959 INFO:teuthology.orchestra.run.smithi092.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 384 kB/s | 25 kB 00:00 2024-03-19T02:50:10.135 INFO:teuthology.orchestra.run.smithi092.stdout:(10/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 8.9 MB/s | 9.8 MB 00:01 2024-03-19T02:50:10.210 INFO:teuthology.orchestra.run.smithi092.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 2.9 MB/s | 221 kB 00:00 2024-03-19T02:50:10.277 INFO:teuthology.orchestra.run.smithi092.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 411 kB/s | 27 kB 00:00 2024-03-19T02:50:10.344 INFO:teuthology.orchestra.run.smithi092.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 479 kB/s | 32 kB 00:00 2024-03-19T02:50:10.515 INFO:teuthology.orchestra.run.smithi092.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 4.3 MB/s | 746 kB 00:00 2024-03-19T02:50:10.595 INFO:teuthology.orchestra.run.smithi092.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 2.0 MB/s | 163 kB 00:00 2024-03-19T02:50:10.695 INFO:teuthology.orchestra.run.smithi092.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.2 MB/s | 127 kB 00:00 2024-03-19T02:50:10.787 INFO:teuthology.orchestra.run.smithi092.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 5.4 MB/s | 501 kB 00:00 2024-03-19T02:50:11.345 INFO:teuthology.orchestra.run.smithi092.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 9.2 MB/s | 5.1 MB 00:00 2024-03-19T02:50:11.412 INFO:teuthology.orchestra.run.smithi092.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 687 kB/s | 45 kB 00:00 2024-03-19T02:50:11.496 INFO:teuthology.orchestra.run.smithi092.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.5 MB/s | 131 kB 00:00 2024-03-19T02:50:11.571 INFO:teuthology.orchestra.run.smithi092.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.1 MB/s | 162 kB 00:00 2024-03-19T02:50:11.688 INFO:teuthology.orchestra.run.smithi092.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 2.7 MB/s | 321 kB 00:00 2024-03-19T02:50:11.772 INFO:teuthology.orchestra.run.smithi092.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.5 MB/s | 298 kB 00:00 2024-03-19T02:50:11.847 INFO:teuthology.orchestra.run.smithi092.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.3 MB/s | 99 kB 00:00 2024-03-19T02:50:11.923 INFO:teuthology.orchestra.run.smithi092.stdout:(25/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-19T02:50:12.523 INFO:teuthology.orchestra.run.smithi092.stdout:(26/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 19 MB/s | 48 MB 00:02 2024-03-19T02:50:12.573 INFO:teuthology.orchestra.run.smithi092.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-19T02:50:12.665 INFO:teuthology.orchestra.run.smithi092.stdout:(28/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 4.2 MB/s | 3.1 MB 00:00 2024-03-19T02:50:12.690 INFO:teuthology.orchestra.run.smithi092.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 292 kB/s | 34 kB 00:00 2024-03-19T02:50:12.716 INFO:teuthology.orchestra.run.smithi092.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.8 MB/s | 139 kB 00:00 2024-03-19T02:50:12.841 INFO:teuthology.orchestra.run.smithi092.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 12 MB/s | 1.8 MB 00:00 2024-03-19T02:50:12.908 INFO:teuthology.orchestra.run.smithi092.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 3.7 MB/s | 248 kB 00:00 2024-03-19T02:50:12.975 INFO:teuthology.orchestra.run.smithi092.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 749 kB/s | 49 kB 00:00 2024-03-19T02:50:13.042 INFO:teuthology.orchestra.run.smithi092.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 250 kB/s | 16 kB 00:00 2024-03-19T02:50:13.117 INFO:teuthology.orchestra.run.smithi092.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.5 MB/s | 264 kB 00:00 2024-03-19T02:50:13.251 INFO:teuthology.orchestra.run.smithi092.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-03-19T02:50:13.301 INFO:teuthology.orchestra.run.smithi092.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 4.0 MB/s | 749 kB 00:00 2024-03-19T02:50:13.326 INFO:teuthology.orchestra.run.smithi092.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 245 kB/s | 18 kB 00:00 2024-03-19T02:50:13.727 INFO:teuthology.orchestra.run.smithi092.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-19T02:50:13.777 INFO:teuthology.orchestra.run.smithi092.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 920 kB 00:00 2024-03-19T02:50:13.802 INFO:teuthology.orchestra.run.smithi092.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-19T02:50:13.878 INFO:teuthology.orchestra.run.smithi092.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 307 kB 00:00 2024-03-19T02:50:13.903 INFO:teuthology.orchestra.run.smithi092.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 317 kB/s | 32 kB 00:00 2024-03-19T02:50:14.070 INFO:teuthology.orchestra.run.smithi092.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 103 kB/s | 17 kB 00:00 2024-03-19T02:50:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.6 MB/s | 3.0 MB 00:00 2024-03-19T02:50:14.354 INFO:teuthology.orchestra.run.smithi092.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 36 kB/s | 10 kB 00:00 2024-03-19T02:50:14.396 INFO:teuthology.orchestra.run.smithi092.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-19T02:50:14.421 INFO:teuthology.orchestra.run.smithi092.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 665 kB/s | 44 kB 00:00 2024-03-19T02:50:14.605 INFO:teuthology.orchestra.run.smithi092.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-19T02:50:14.638 INFO:teuthology.orchestra.run.smithi092.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 2.6 MB/s | 578 kB 00:00 2024-03-19T02:50:14.689 INFO:teuthology.orchestra.run.smithi092.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 576 kB/s | 46 kB 00:00 2024-03-19T02:50:14.731 INFO:teuthology.orchestra.run.smithi092.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-19T02:50:14.765 INFO:teuthology.orchestra.run.smithi092.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-19T02:50:14.815 INFO:teuthology.orchestra.run.smithi092.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-19T02:50:14.840 INFO:teuthology.orchestra.run.smithi092.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 457 kB/s | 34 kB 00:00 2024-03-19T02:50:15.366 INFO:teuthology.orchestra.run.smithi092.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 2.9 MB/s | 1.5 MB 00:00 2024-03-19T02:50:15.466 INFO:teuthology.orchestra.run.smithi092.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 7.3 MB/s | 4.7 MB 00:00 2024-03-19T02:50:15.508 INFO:teuthology.orchestra.run.smithi092.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 5.9 MB/s | 811 kB 00:00 2024-03-19T02:50:15.533 INFO:teuthology.orchestra.run.smithi092.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-19T02:50:15.925 INFO:teuthology.orchestra.run.smithi092.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 14 MB/s | 5.7 MB 00:00 2024-03-19T02:50:15.967 INFO:teuthology.orchestra.run.smithi092.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-03-19T02:50:16.017 INFO:teuthology.orchestra.run.smithi092.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-19T02:50:16.301 INFO:teuthology.orchestra.run.smithi092.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-19T02:50:16.326 INFO:teuthology.orchestra.run.smithi092.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 81 kB/s | 24 kB 00:00 2024-03-19T02:50:16.351 INFO:teuthology.orchestra.run.smithi092.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 369 kB/s | 18 kB 00:00 2024-03-19T02:50:16.399 INFO:teuthology.orchestra.run.smithi092.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 144 kB 00:00 2024-03-19T02:50:16.435 INFO:teuthology.orchestra.run.smithi092.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 387 kB/s | 32 kB 00:00 2024-03-19T02:50:16.627 INFO:teuthology.orchestra.run.smithi092.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-19T02:50:16.652 INFO:teuthology.orchestra.run.smithi092.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-19T02:50:16.678 INFO:teuthology.orchestra.run.smithi092.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.5 MB/s | 174 kB 00:00 2024-03-19T02:50:16.703 INFO:teuthology.orchestra.run.smithi092.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-19T02:50:16.728 INFO:teuthology.orchestra.run.smithi092.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 836 kB/s | 41 kB 00:00 2024-03-19T02:50:16.826 INFO:teuthology.orchestra.run.smithi092.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 238 kB 00:00 2024-03-19T02:50:16.854 INFO:teuthology.orchestra.run.smithi092.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-19T02:50:16.888 INFO:teuthology.orchestra.run.smithi092.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.4 MB/s | 211 kB 00:00 2024-03-19T02:50:16.913 INFO:teuthology.orchestra.run.smithi092.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 846 kB/s | 47 kB 00:00 2024-03-19T02:50:16.939 INFO:teuthology.orchestra.run.smithi092.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 876 kB/s | 43 kB 00:00 2024-03-19T02:50:17.147 INFO:teuthology.orchestra.run.smithi092.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 206 kB/s | 39 kB 00:00 2024-03-19T02:50:17.431 INFO:teuthology.orchestra.run.smithi092.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-19T02:50:17.456 INFO:teuthology.orchestra.run.smithi092.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 139 kB/s | 41 kB 00:00 2024-03-19T02:50:17.498 INFO:teuthology.orchestra.run.smithi092.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 56 kB 00:00 2024-03-19T02:50:17.563 INFO:teuthology.orchestra.run.smithi092.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 300 kB 00:00 2024-03-19T02:50:17.599 INFO:teuthology.orchestra.run.smithi092.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 644 kB/s | 63 kB 00:00 2024-03-19T02:50:17.624 INFO:teuthology.orchestra.run.smithi092.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 737 kB/s | 44 kB 00:00 2024-03-19T02:50:17.649 INFO:teuthology.orchestra.run.smithi092.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-19T02:50:17.683 INFO:teuthology.orchestra.run.smithi092.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-19T02:50:17.742 INFO:teuthology.orchestra.run.smithi092.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-19T02:50:17.767 INFO:teuthology.orchestra.run.smithi092.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-19T02:50:17.784 INFO:teuthology.orchestra.run.smithi092.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2024-03-19T02:50:17.818 INFO:teuthology.orchestra.run.smithi092.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.8 MB/s | 187 kB 00:00 2024-03-19T02:50:26.102 INFO:teuthology.orchestra.run.smithi092.stdout:(91/174): ceph-common-19.0.0-2206.gb6ae2491.el9 1.1 MB/s | 21 MB 00:18 2024-03-19T02:52:31.904 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] jq-1.6-16.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/jq-1.6-16.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:52:31.905 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:52:32.435 INFO:teuthology.orchestra.run.smithi092.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.4 kB/s | 187 kB 02:14 2024-03-19T02:52:32.635 INFO:teuthology.orchestra.run.smithi092.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 626 kB/s | 125 kB 00:00 2024-03-19T02:52:32.718 INFO:teuthology.orchestra.run.smithi092.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 260 kB/s | 21 kB 00:00 2024-03-19T02:52:32.777 INFO:teuthology.orchestra.run.smithi092.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 438 kB/s | 25 kB 00:00 2024-03-19T02:52:32.994 INFO:teuthology.orchestra.run.smithi092.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 19 MB/s | 4.1 MB 00:00 2024-03-19T02:52:33.028 INFO:teuthology.orchestra.run.smithi092.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 kB/s | 803 kB 02:15 2024-03-19T02:52:33.062 INFO:teuthology.orchestra.run.smithi092.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-19T02:52:33.112 INFO:teuthology.orchestra.run.smithi092.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.3 MB/s | 216 kB 00:00 2024-03-19T02:52:33.154 INFO:teuthology.orchestra.run.smithi092.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 849 kB/s | 35 kB 00:00 2024-03-19T02:52:33.187 INFO:teuthology.orchestra.run.smithi092.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-03-19T02:52:33.204 INFO:teuthology.orchestra.run.smithi092.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 898 kB/s | 44 kB 00:00 2024-03-19T02:52:33.238 INFO:teuthology.orchestra.run.smithi092.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-19T02:52:33.493 INFO:teuthology.orchestra.run.smithi092.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 77 kB/s | 22 kB 00:00 2024-03-19T02:52:34.097 INFO:teuthology.orchestra.run.smithi092.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-03-19T02:52:34.130 INFO:teuthology.orchestra.run.smithi092.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 861 kB/s | 548 kB 00:00 2024-03-19T02:52:34.239 INFO:teuthology.orchestra.run.smithi092.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-03-19T02:52:34.664 INFO:teuthology.orchestra.run.smithi092.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-19T02:52:34.698 INFO:teuthology.orchestra.run.smithi092.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 515 kB/s | 309 kB 00:00 2024-03-19T02:52:34.748 INFO:teuthology.orchestra.run.smithi092.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-03-19T02:52:34.799 INFO:teuthology.orchestra.run.smithi092.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-19T02:52:34.849 INFO:teuthology.orchestra.run.smithi092.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-19T02:52:34.916 INFO:teuthology.orchestra.run.smithi092.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-19T02:52:34.974 INFO:teuthology.orchestra.run.smithi092.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-03-19T02:52:35.075 INFO:teuthology.orchestra.run.smithi092.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 436 kB/s | 43 kB 00:00 2024-03-19T02:52:35.108 INFO:teuthology.orchestra.run.smithi092.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-19T02:52:35.175 INFO:teuthology.orchestra.run.smithi092.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 323 kB/s | 32 kB 00:00 2024-03-19T02:52:35.225 INFO:teuthology.orchestra.run.smithi092.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-03-19T02:52:35.276 INFO:teuthology.orchestra.run.smithi092.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-19T02:52:35.359 INFO:teuthology.orchestra.run.smithi092.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-19T02:52:35.460 INFO:teuthology.orchestra.run.smithi092.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-03-19T02:52:35.485 INFO:teuthology.orchestra.run.smithi092.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 283 kB/s | 35 kB 00:00 2024-03-19T02:52:35.577 INFO:teuthology.orchestra.run.smithi092.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-19T02:52:35.702 INFO:teuthology.orchestra.run.smithi092.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-19T02:52:35.761 INFO:teuthology.orchestra.run.smithi092.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-03-19T02:52:35.820 INFO:teuthology.orchestra.run.smithi092.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-19T02:52:35.862 INFO:teuthology.orchestra.run.smithi092.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-03-19T02:52:35.920 INFO:teuthology.orchestra.run.smithi092.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-03-19T02:52:35.970 INFO:teuthology.orchestra.run.smithi092.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-03-19T02:52:36.021 INFO:teuthology.orchestra.run.smithi092.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-03-19T02:52:36.129 INFO:teuthology.orchestra.run.smithi092.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-03-19T02:52:36.155 INFO:teuthology.orchestra.run.smithi092.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 348 kB/s | 46 kB 00:00 2024-03-19T02:52:36.247 INFO:teuthology.orchestra.run.smithi092.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 678 kB/s | 79 kB 00:00 2024-03-19T02:52:36.275 INFO:teuthology.orchestra.run.smithi092.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 482 kB/s | 58 kB 00:00 2024-03-19T02:52:36.384 INFO:teuthology.orchestra.run.smithi092.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-19T02:52:36.401 INFO:teuthology.orchestra.run.smithi092.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-19T02:52:36.484 INFO:teuthology.orchestra.run.smithi092.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 903 kB/s | 90 kB 00:00 2024-03-19T02:52:36.510 INFO:teuthology.orchestra.run.smithi092.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-03-19T02:52:36.576 INFO:teuthology.orchestra.run.smithi092.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 337 kB/s | 31 kB 00:00 2024-03-19T02:52:36.610 INFO:teuthology.orchestra.run.smithi092.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-19T02:52:36.669 INFO:teuthology.orchestra.run.smithi092.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 646 kB/s | 59 kB 00:00 2024-03-19T02:52:36.702 INFO:teuthology.orchestra.run.smithi092.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 391 kB/s | 36 kB 00:00 2024-03-19T02:52:36.761 INFO:teuthology.orchestra.run.smithi092.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 557 kB/s | 51 kB 00:00 2024-03-19T02:52:36.812 INFO:teuthology.orchestra.run.smithi092.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-19T02:52:36.862 INFO:teuthology.orchestra.run.smithi092.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 896 kB/s | 90 kB 00:00 2024-03-19T02:52:36.962 INFO:teuthology.orchestra.run.smithi092.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-19T02:52:36.996 INFO:teuthology.orchestra.run.smithi092.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-19T02:52:37.063 INFO:teuthology.orchestra.run.smithi092.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-19T02:52:37.155 INFO:teuthology.orchestra.run.smithi092.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-19T02:52:37.697 INFO:teuthology.orchestra.run.smithi092.stdout:(150/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 6.6 MB/s | 3.1 MB 00:00 2024-03-19T02:52:37.922 INFO:teuthology.orchestra.run.smithi092.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.4 MB/s | 321 kB 00:00 2024-03-19T02:52:38.047 INFO:teuthology.orchestra.run.smithi092.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 3.4 MB/s | 3.4 MB 00:00 2024-03-19T02:52:39.599 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:52:39.889 INFO:teuthology.orchestra.run.smithi092.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.0 MB/s | 1.8 MB 00:01 2024-03-19T02:52:39.997 INFO:teuthology.orchestra.run.smithi092.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 391 kB/s | 42 kB 00:00 2024-03-19T02:52:40.131 INFO:teuthology.orchestra.run.smithi092.stdout:(155/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:14 2024-03-19T02:52:40.198 INFO:teuthology.orchestra.run.smithi092.stdout:(156/174): NetworkManager-1.47.2-1.el9.x86_64.r 1.0 MB/s | 2.3 MB 00:02 2024-03-19T02:52:40.232 INFO:teuthology.orchestra.run.smithi092.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.0 MB/s | 248 kB 00:00 2024-03-19T02:52:40.315 INFO:teuthology.orchestra.run.smithi092.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 313 kB/s | 36 kB 00:00 2024-03-19T02:52:40.432 INFO:teuthology.orchestra.run.smithi092.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 962 kB/s | 192 kB 00:00 2024-03-19T02:52:40.541 INFO:teuthology.orchestra.run.smithi092.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 214 kB/s | 23 kB 00:00 2024-03-19T02:52:40.633 INFO:teuthology.orchestra.run.smithi092.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 807 kB/s | 256 kB 00:00 2024-03-19T02:52:40.658 INFO:teuthology.orchestra.run.smithi092.stdout:(162/174): acl-2.3.1-4.el9.x86_64.rpm 135 kB/s | 71 kB 00:00 2024-03-19T02:52:40.683 INFO:teuthology.orchestra.run.smithi092.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 765 kB/s | 108 kB 00:00 2024-03-19T02:52:40.792 INFO:teuthology.orchestra.run.smithi092.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 592 kB/s | 64 kB 00:00 2024-03-19T02:52:40.825 INFO:teuthology.orchestra.run.smithi092.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 807 kB/s | 155 kB 00:00 2024-03-19T02:52:40.859 INFO:teuthology.orchestra.run.smithi092.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 678 kB/s | 136 kB 00:00 2024-03-19T02:52:40.901 INFO:teuthology.orchestra.run.smithi092.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 264 kB/s | 29 kB 00:00 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi092.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:01 2024-03-19T02:52:42.576 INFO:teuthology.orchestra.run.smithi092.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 642 kB/s | 465 kB 00:00 2024-03-19T02:52:42.901 INFO:teuthology.orchestra.run.smithi092.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.1 MB/s | 2.2 MB 00:02 2024-03-19T02:52:43.085 INFO:teuthology.orchestra.run.smithi092.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 2.3 MB 00:02 2024-03-19T02:52:43.177 INFO:teuthology.orchestra.run.smithi092.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 812 kB/s | 74 kB 00:00 2024-03-19T02:52:43.302 INFO:teuthology.orchestra.run.smithi092.stdout:(173/174): util-linux-user-2.37.4-18.el9.x86_64 43 kB/s | 31 kB 00:00 2024-03-19T02:52:43.719 INFO:teuthology.orchestra.run.smithi092.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 1.3 MB/s | 1.1 MB 00:00 2024-03-19T02:52:43.727 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:43.728 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.4 MB/s | 224 MB 02:37 2024-03-19T02:52:45.157 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-19T02:52:45.500 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-19T02:52:45.500 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-19T02:52:47.920 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-19T02:52:47.921 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-19T02:52:55.300 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:52:55.341 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:52:55.404 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:52:55.447 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:52:55.534 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:52:55.634 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:52:55.687 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:52:55.719 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:52:55.800 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:52:56.209 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:52:56.242 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:52:56.391 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:52:56.427 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:52:56.570 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:52:56.777 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:52:56.854 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:52:57.202 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:52:57.257 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:52:57.282 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:52:57.436 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:52:57.466 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:52:57.602 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:52:57.635 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:52:57.696 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:52:58.001 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:52:58.051 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:52:58.102 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:52:58.484 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:52:58.636 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:52:58.734 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:52:58.947 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:52:58.997 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:52:59.342 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:52:59.494 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:52:59.604 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:52:59.645 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:52:59.680 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:52:59.739 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:52:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:52:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:52:59.962 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:53:00.070 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:53:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:53:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:53:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:53:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:53:00.371 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:53:00.406 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:53:00.456 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:53:00.487 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:53:00.598 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:53:00.658 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:53:00.747 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:53:00.927 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:53:00.993 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:53:01.681 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:53:01.799 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:53:01.939 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:53:02.000 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:53:02.027 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:53:02.076 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:53:02.146 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:53:02.202 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:53:02.250 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:53:02.708 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:53:02.903 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:53:02.996 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:53:03.249 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:53:03.300 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:53:03.377 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:53:03.468 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:53:03.508 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:53:03.546 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:53:03.613 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:53:03.655 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:53:03.724 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:53:03.787 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:53:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:53:04.061 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:53:04.099 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:53:04.141 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:53:04.274 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:53:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:53:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:53:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:53:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:53:04.567 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:53:04.753 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:53:04.893 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:53:04.940 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:53:05.001 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:53:05.077 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:53:05.321 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:53:05.376 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:53:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:53:05.589 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:53:05.630 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:53:05.660 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:53:05.700 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:53:05.758 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:53:05.834 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:53:05.891 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:53:05.941 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:53:06.066 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:53:06.101 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:53:06.136 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:53:06.187 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:53:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:53:06.371 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:53:06.484 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:53:06.553 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:53:06.589 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:53:06.645 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:53:06.783 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:53:06.853 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:53:07.837 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:53:07.894 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:53:07.928 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:53:07.986 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:53:08.097 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:53:08.154 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:53:08.426 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:53:08.508 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:53:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:53:09.437 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:53:10.974 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:53:11.071 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:53:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:53:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:53:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:53:11.549 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:53:11.586 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:53:11.706 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:53:11.742 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:53:11.796 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:53:12.306 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:53:12.339 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:53:12.413 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:53:12.441 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:53:15.004 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:53:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:53:15.685 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:53:15.724 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:53:15.872 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:53:15.872 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:53:15.872 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:53:15.872 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:53:29.033 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:53:29.033 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-03-19T02:53:29.033 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-03-19T02:53:29.033 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-03-19T02:53:29.034 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-03-19T02:53:29.034 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-03-19T02:53:29.034 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-03-19T02:53:29.034 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-03-19T02:53:29.034 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:29.254 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:53:29.302 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:53:29.302 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:29.302 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:53:29.302 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:30.888 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:53:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:53:30.969 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:30.970 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:53:30.970 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:53:30.970 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:53:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:31.205 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:53:31.274 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:53:31.275 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:31.275 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:53:31.275 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:53:31.275 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:53:31.275 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:53:31.841 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:31.986 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:53:32.633 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:53:32.676 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:53:32.745 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:53:32.770 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:53:33.088 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:53:33.142 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:53:34.247 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:53:34.306 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:53:34.453 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:53:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:53:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:53:34.751 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:53:34.751 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:34.752 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:53:34.752 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:53:34.752 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:53:34.752 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:34.808 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:53:34.956 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:53:35.903 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:53:35.935 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:53:36.030 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:53:36.030 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:36.030 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:53:36.031 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:53:36.031 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:53:36.031 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:36.085 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:53:36.179 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:53:36.180 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:36.180 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:53:36.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:36.269 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:53:36.374 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:36.770 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:53:36.865 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:41.658 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:53:41.702 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:53:41.742 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:53:41.818 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:53:42.101 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:53:42.226 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:53:42.282 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:53:42.335 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:53:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:53:42.470 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:53:42.505 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:53:42.551 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:53:42.602 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:53:42.687 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:53:42.805 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:53:42.850 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:53:43.748 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:53:43.789 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:53:43.845 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:53:45.162 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:53:45.294 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:53:45.349 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:53:45.491 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:53:45.529 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:53:45.562 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:53:45.597 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:53:45.647 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:53:45.647 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:53:45.792 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:53:45.842 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:53:45.842 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:53:45.964 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:53:46.008 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:53:46.055 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:53:46.087 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:53:46.119 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:53:46.185 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:53:46.258 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:53:46.299 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:53:46.333 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:53:46.396 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:53:46.433 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:53:46.467 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:53:46.504 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:53:46.504 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:53:46.833 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:53:46.833 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:53:46.997 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:53:47.027 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:53:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:53:47.113 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:53:47.154 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:53:47.273 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:53:47.527 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:53:47.527 INFO:teuthology.orchestra.run.smithi092.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-19T02:53:47.527 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:52.674 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:53:52.675 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:53:52.676 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:53:52.692 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:53:52.693 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:53:52.694 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:53:52.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:53:52.696 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:53:52.697 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:53:52.698 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:53:52.699 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:53:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:53:52.701 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:53:52.702 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:53:52.703 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:53:52.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:53:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:53:55.239 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:53:55.240 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:53:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.242 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.243 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.244 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:53:55.245 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:53:55.246 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:53:55.247 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:53:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.249 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:53:55.250 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:53:55.251 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:53:55.252 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:53:55.252 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:53:55.252 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-19T02:53:55.882 DEBUG:teuthology.parallel:result is None 2024-03-19T02:53:55.883 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:53:56.100 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:53:56.155 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:53:56.156 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:53:56.156 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:53:56.158 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:53:56.372 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:53:56.398 INFO:teuthology.orchestra.run.smithi159.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:53:56.399 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:53:56.399 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:53:56.400 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T02:53:56.401 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:56.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:53:56.494 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:56.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:53:56.530 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T02:53:56.530 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:56.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:53:56.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:53:56.724 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:56.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:53:56.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:53:56.827 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T02:53:56.827 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:56.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:53:56.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:53:57.015 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:57.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:53:57.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:53:57.110 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T02:53:57.111 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:57.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:53:57.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:53:57.286 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:57.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:53:57.313 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:53:57.379 INFO:teuthology.run_tasks:Running task ceph... 2024-03-19T02:53:57.462 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-19T02:53:57.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2024-03-19T02:53:57.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-03-19T02:53:57.534 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-19T02:53:57.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T02:53:57.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T02:53:57.606 INFO:tasks.ceph:Creating extra log directories... 2024-03-19T02:53:57.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T02:53:57.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T02:53:57.712 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-19T02:53:57.713 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-19T02:53:57.713 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-19T02:53:57.713 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:57.713 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T02:53:57.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T02:53:57.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-19T02:53:58.020 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:58.020 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T02:53:58.035 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T02:53:58.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-19T02:53:58.185 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-19T02:53:58.185 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-19T02:53:58.185 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441', 'branch': 'wip-batrick-testing-20240318.181317', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7609441', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240318.181317'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240318.181317', 'sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3005, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'sleep_before_teardown': 0, 'subset': '433/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240318.181317', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFx719IkU9FJAu59YtRWy7dpjI33NS5tXulL6SGSomLxamg9kPlqlUVUqWY0X+kfcWkNNlTabPYR7Zi+Zj+FNo=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDa8khp7bshqgwJj1KEBwKqaixzbd4GAw47cz1EWzErQ2TUConfFiFw4/FY8kSd10LsmnQXHr5Lw0wyvVfMeYkA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-18_21:05:49', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-19T02:53:58.186 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T02:53:58.191 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T02:53:58.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T02:53:58.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T02:53:58.339 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:53:58.339 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:53:58.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:53:58.377 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-03-19T02:53:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-19 02:53:47.980739026 +0000 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-19 02:43:30.124241315 +0000 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-19 02:43:30.124241315 +0000 2024-03-19T02:53:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-19 02:43:30.124241315 +0000 2024-03-19T02:53:58.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:53:58.569 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-19T02:53:58.569 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-19T02:53:58.569 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000373857 s, 1.4 MB/s 2024-03-19T02:53:58.572 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:53:58.612 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-03-19T02:53:58.696 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:53:58.696 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:58.696 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 655 Links: 1 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-19 02:53:47.981738997 +0000 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-19 02:43:30.696224976 +0000 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-19 02:43:30.696224976 +0000 2024-03-19T02:53:58.697 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-19 02:43:30.696224976 +0000 2024-03-19T02:53:58.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:53:58.802 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-19T02:53:58.802 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-19T02:53:58.802 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000257311 s, 2.0 MB/s 2024-03-19T02:53:58.804 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:53:58.848 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-19 02:53:47.981738997 +0000 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-19 02:43:31.247209235 +0000 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-19 02:43:31.247209235 +0000 2024-03-19T02:53:58.931 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-19 02:43:31.247209235 +0000 2024-03-19T02:53:58.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:53:59.039 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-19T02:53:59.040 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-19T02:53:59.040 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000278704 s, 1.8 MB/s 2024-03-19T02:53:59.044 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:53:59.085 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-19 02:53:47.982738968 +0000 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-19 02:43:31.827192666 +0000 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-19 02:43:31.827192666 +0000 2024-03-19T02:53:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-19 02:43:31.827192666 +0000 2024-03-19T02:53:59.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:53:59.269 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-19T02:53:59.269 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-19T02:53:59.270 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000287574 s, 1.8 MB/s 2024-03-19T02:53:59.272 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:53:59.315 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T02:53:59.316 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:53:59.316 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:53:59.334 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:53:59.334 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-19 02:46:36.737714119 +0000 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-19 02:44:32.361090269 +0000 2024-03-19T02:53:59.392 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-19 02:44:32.361090269 +0000 2024-03-19T02:53:59.393 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-19 02:44:32.361090269 +0000 2024-03-19T02:53:59.393 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:53:59.461 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-19T02:53:59.462 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-19T02:53:59.462 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000261917 s, 2.0 MB/s 2024-03-19T02:53:59.463 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:53:59.521 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-03-19T02:53:59.578 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:53:59.578 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:59.578 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-19 02:46:36.737714119 +0000 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-19 02:44:32.708083573 +0000 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-19 02:44:32.708083573 +0000 2024-03-19T02:53:59.579 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-19 02:44:32.708083573 +0000 2024-03-19T02:53:59.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:53:59.647 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-19T02:53:59.647 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-19T02:53:59.647 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000210597 s, 2.4 MB/s 2024-03-19T02:53:59.648 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:53:59.706 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-19 02:46:36.737714119 +0000 2024-03-19T02:53:59.762 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-19 02:44:33.051076955 +0000 2024-03-19T02:53:59.763 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-19 02:44:33.051076955 +0000 2024-03-19T02:53:59.763 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-19 02:44:33.051076955 +0000 2024-03-19T02:53:59.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:53:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-19T02:53:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-19T02:53:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000222798 s, 2.3 MB/s 2024-03-19T02:53:59.832 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:53:59.890 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-19 02:46:36.738714100 +0000 2024-03-19T02:53:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-19 02:44:33.400070221 +0000 2024-03-19T02:53:59.947 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-19 02:44:33.400070221 +0000 2024-03-19T02:53:59.947 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-19 02:44:33.400070221 +0000 2024-03-19T02:53:59.947 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:54:00.017 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-19T02:54:00.017 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-19T02:54:00.018 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000172406 s, 3.0 MB/s 2024-03-19T02:54:00.019 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:54:00.076 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:00.076 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T02:54:00.076 INFO:tasks.ceph:Generating config... 2024-03-19T02:54:00.077 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-19T02:54:00.077 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-03-19T02:54:00.078 INFO:tasks.ceph:[client] debug client = 20 2024-03-19T02:54:00.078 INFO:tasks.ceph:[client] debug ms = 1 2024-03-19T02:54:00.078 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-19T02:54:00.078 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-19T02:54:00.078 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-19T02:54:00.078 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-19T02:54:00.079 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-19T02:54:00.080 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-19T02:54:00.080 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-19T02:54:00.080 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-19T02:54:00.080 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-19T02:54:00.080 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-19T02:54:00.081 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-19T02:54:00.081 INFO:tasks.ceph:Setting up mon.a... 2024-03-19T02:54:00.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-19T02:54:00.221 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.keyring 2024-03-19T02:54:00.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-19T02:54:00.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T02:54:00.446 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.92'), ('mon.b', '172.21.15.159'), ('mon.c', '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]')] 2024-03-19T02:54:00.447 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.92,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-19T02:54:00.448 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:00.448 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T02:54:00.498 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.92 --add b 172.21.15.159 --addv c '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: generated fsid 91932042-1f9e-4210-9e82-70c11e6277fd 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:setting min_mon_release = quincy 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:epoch 0 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:fsid 91932042-1f9e-4210-9e82-70c11e6277fd 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:last_changed 2024-03-19T02:54:00.630021+0000 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:created 2024-03-19T02:54:00.630021+0000 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:min_mon_release 17 (quincy) 2024-03-19T02:54:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:election_strategy: 1 2024-03-19T02:54:00.631 INFO:teuthology.orchestra.run.smithi092.stdout:0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-03-19T02:54:00.631 INFO:teuthology.orchestra.run.smithi092.stdout:1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.b 2024-03-19T02:54:00.631 INFO:teuthology.orchestra.run.smithi092.stdout:2: [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] mon.c 2024-03-19T02:54:00.631 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-19T02:54:00.633 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T02:54:00.674 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 91932042-1f9e-4210-9e82-70c11e6277fd... 2024-03-19T02:54:00.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T02:54:00.721 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-03-19T02:54:00.779 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-03-19T02:54:00.780 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-03-19T02:54:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.92,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on skipped message = False 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 91932042-1f9e-4210-9e82-70c11e6277fd 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.782 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression mode = aggressive 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: mon client directed command retry = 5 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-03-19T02:54:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay max = 1 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay probability = 0.005 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay type = client mds 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject socket failures = 2500 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: osd blocked scrub grace period = 3600 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: osd max markdown count = 1000 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-19T02:54:00.785 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-03-19T02:54:00.786 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2024-03-19T02:54:00.787 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: client_shutdown_timeout = 120 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-03-19T02:54:00.788 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.e] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.f] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2024-03-19T02:54:00.789 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds balancer = 20 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal fragment size max = 10000 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal merge size = 5 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split bits = 3 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split size = 100 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds oft prefetch dirfrags = False 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds_dir_prefetch = False 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mds_sleep_rank_change = 5000000.0 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: mon client directed command retry = 5 2024-03-19T02:54:00.790 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay max = 1 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay probability = 0.005 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay type = client mds osd 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject socket failures = 2500 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-03-19T02:54:00.791 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-03-19T02:54:00.880 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-03-19T02:54:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2024-03-19T02:54:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T02:54:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2024-03-19T02:54:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T02:54:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2024-03-19T02:54:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.92,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on skipped message = False 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 91932042-1f9e-4210-9e82-70c11e6277fd 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2024-03-19T02:54:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug trim objects = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2024-03-19T02:54:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore compression mode = aggressive 2024-03-19T02:54:00.885 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 20 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 20 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: mon client directed command retry = 5 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay max = 1 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay probability = 0.005 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay type = client mds 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject socket failures = 2500 2024-03-19T02:54:00.886 INFO:teuthology.orchestra.run.smithi092.stdout: osd blocked scrub grace period = 3600 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: osd max markdown count = 1000 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.887 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T02:54:00.888 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2024-03-19T02:54:00.889 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: client_shutdown_timeout = 120 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.e] 2024-03-19T02:54:00.890 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.f] 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds balancer = 20 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal fragment size max = 10000 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal merge size = 5 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split bits = 3 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split size = 100 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds oft prefetch dirfrags = False 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2024-03-19T02:54:00.891 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: mds_dir_prefetch = False 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: mds_sleep_rank_change = 5000000.0 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: mon client directed command retry = 5 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay max = 1 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay probability = 0.005 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay type = client mds osd 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject socket failures = 2500 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-19T02:54:00.892 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-19T02:54:00.924 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-19T02:54:00.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-19T02:54:01.066 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-19T02:54:01.066 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:01.066 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-19T02:54:01.116 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:01.116 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-19T02:54:01.167 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:54:01.167 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:01.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T02:54:01.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T02:54:01.305 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:01.305 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:01.401 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:54:01.401 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:01.401 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T02:54:01.401 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T02:54:01.446 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:01.446 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:01.503 INFO:tasks.ceph:Setting up mon nodes... 2024-03-19T02:54:01.503 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-19T02:54:01.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T02:54:01.690 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T02:54:01.700 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T02:54:01.771 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T02:54:01.774 INFO:tasks.ceph:Setting up mds nodes... 2024-03-19T02:54:01.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T02:54:01.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-19T02:54:02.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-19T02:54:02.225 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-19T02:54:02.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-19T02:54:02.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-19T02:54:02.495 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-19T02:54:02.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-19T02:54:02.593 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T02:54:02.655 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T02:54:02.657 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-19T02:54:02.686 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-19T02:54:02.783 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-19T02:54:02.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-19T02:54:02.813 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-19T02:54:02.907 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-19T02:54:02.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-19T02:54:02.936 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-19T02:54:02.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T02:54:03.066 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-19T02:54:03.121 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-19T02:54:03.121 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T02:54:03.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-19T02:54:03.211 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:03.212 INFO:tasks.ceph:role: osd.0 2024-03-19T02:54:03.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:54:03.212 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:03.833 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-19T02:54:03.840 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-19T02:54:03.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-19T02:54:04.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-19T02:54:04.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-19T02:54:04.243 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:04.243 INFO:tasks.ceph:role: osd.1 2024-03-19T02:54:04.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:54:04.243 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:04.341 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:04.822 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-19T02:54:04.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-19T02:54:04.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-19T02:54:04.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-19T02:54:05.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-19T02:54:05.154 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:05.154 INFO:tasks.ceph:role: osd.2 2024-03-19T02:54:05.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:54:05.154 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T02:54:05.253 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:05.253 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:05.253 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:05.253 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:05.254 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-19T02:54:05.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-19T02:54:05.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-19T02:54:05.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-19T02:54:06.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-19T02:54:06.131 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:06.131 INFO:tasks.ceph:role: osd.3 2024-03-19T02:54:06.131 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T02:54:06.131 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:06.229 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:06.686 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-19T02:54:06.691 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-19T02:54:06.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-19T02:54:06.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-19T02:54:06.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:07.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.069+0000 7fca2f015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-19T02:54:07.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.070+0000 7fca2f015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-19T02:54:07.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.070+0000 7fca2f015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T02:54:07.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.070+0000 7fca2f015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T02:54:07.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.070+0000 7fca2f015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T02:54:07.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:07.076+0000 7fca2f015640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-19T02:54:10.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-19T02:54:10.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:10.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.239+0000 7f7afe1bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-19T02:54:10.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.239+0000 7f7afe1bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-19T02:54:10.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.240+0000 7f7afe1bd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T02:54:10.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.240+0000 7f7afe1bd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T02:54:10.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.240+0000 7f7afe1bd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T02:54:10.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:10.242+0000 7f7afe1bd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-19T02:54:13.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-19T02:54:13.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:13.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.625+0000 7f6153bff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-19T02:54:13.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.625+0000 7f6153bff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-19T02:54:13.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.626+0000 7f6153bff640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T02:54:13.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.626+0000 7f6153bff640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T02:54:13.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.626+0000 7f6153bff640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T02:54:13.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:13.628+0000 7f6153bff640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-19T02:54:16.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-19T02:54:16.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:16.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.798+0000 7fa8a8015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-19T02:54:16.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.799+0000 7fa8a8015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-19T02:54:16.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.799+0000 7fa8a8015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T02:54:16.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.799+0000 7fa8a8015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T02:54:16.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.799+0000 7fa8a8015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T02:54:16.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:16.801+0000 7fa8a8015640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-19T02:54:19.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-19T02:54:19.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-19T02:54:19.860 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:19.860 INFO:tasks.ceph:role: osd.4 2024-03-19T02:54:19.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:54:19.860 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:19.943 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:19.944 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:20.427 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-19T02:54:20.429 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-19T02:54:20.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-19T02:54:20.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-19T02:54:20.607 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-19T02:54:20.675 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:20.675 INFO:tasks.ceph:role: osd.5 2024-03-19T02:54:20.675 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:54:20.675 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T02:54:20.751 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:20.752 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:21.214 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-19T02:54:21.216 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-19T02:54:21.216 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-19T02:54:21.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-19T02:54:21.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-19T02:54:21.407 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:21.407 INFO:tasks.ceph:role: osd.6 2024-03-19T02:54:21.408 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:54:21.408 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T02:54:21.482 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:21.482 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:21.483 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:21.970 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-19T02:54:21.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-19T02:54:21.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-19T02:54:22.115 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-19T02:54:22.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-19T02:54:22.217 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T02:54:22.217 INFO:tasks.ceph:role: osd.7 2024-03-19T02:54:22.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T02:54:22.218 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T02:54:22.293 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T02:54:22.293 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T02:54:22.293 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T02:54:22.293 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-19T02:54:22.293 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T02:54:22.294 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-19T02:54:22.294 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T02:54:22.294 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-19T02:54:22.294 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T02:54:22.294 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T02:54:22.816 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-19T02:54:22.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-19T02:54:22.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-19T02:54:22.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-19T02:54:22.980 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:23.086 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.085+0000 7f94ffb11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-19T02:54:23.086 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.085+0000 7f94ffb11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-19T02:54:23.087 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.085+0000 7f94ffb11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T02:54:23.087 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.085+0000 7f94ffb11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T02:54:23.087 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.085+0000 7f94ffb11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T02:54:23.087 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:23.086+0000 7f94ffb11640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-19T02:54:25.943 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-19T02:54:25.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:26.065 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.064+0000 7ff3e0a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-19T02:54:26.066 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.064+0000 7ff3e0a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-19T02:54:26.066 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.064+0000 7ff3e0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T02:54:26.066 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.064+0000 7ff3e0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T02:54:26.066 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.064+0000 7ff3e0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T02:54:26.066 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:26.065+0000 7ff3e0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-19T02:54:28.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:54:28.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:54:28.906 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-19T02:54:28.935 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:29.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.024+0000 7f5b0c01f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-19T02:54:29.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.024+0000 7f5b0c01f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-19T02:54:29.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.024+0000 7f5b0c01f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T02:54:29.025 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.024+0000 7f5b0c01f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T02:54:29.026 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.024+0000 7f5b0c01f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T02:54:29.026 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:29.025+0000 7f5b0c01f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-19T02:54:31.879 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-19T02:54:31.909 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:32.001 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.000+0000 7f0e10811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-19T02:54:32.001 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.000+0000 7f0e10811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-19T02:54:32.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.000+0000 7f0e10811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T02:54:32.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.000+0000 7f0e10811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T02:54:32.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.000+0000 7f0e10811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T02:54:32.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:32.001+0000 7f0e10811640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-19T02:54:34.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-19T02:54:34.880 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-19T02:54:34.880 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:34.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-19T02:54:34.982 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:34.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-19T02:54:35.076 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-19T02:54:35.170 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-19T02:54:35.261 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-19T02:54:35.355 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-19T02:54:35.445 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-19T02:54:35.541 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:35.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-19T02:54:35.637 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.637 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-19T02:54:35.669 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-19T02:54:35.738 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-19T02:54:35.807 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-19T02:54:35.874 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.874 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-19T02:54:35.941 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:35.942 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-19T02:54:36.008 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:36.008 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-19T02:54:36.075 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:36.075 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-19T02:54:36.142 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:36.142 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-19T02:54:36.197 INFO:tasks.ceph:Adding keys to all mons... 2024-03-19T02:54:36.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T02:54:36.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBJ/vhleUcgKRAAI4pxJ1Fs/nJrQjhSmo4arg== 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBJ/vhl4j2pOBAAxAFwzNbUjM4AP5bCCFjHeg== 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBK/vhl3shyDRAAOEJMRSRo49bo/kSEhvekKQ== 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.e] 2024-03-19T02:54:36.227 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBK/vhlcUGPHRAAzpj7VtfXwV3nNS2G/fDRUQ== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBP/vhlCB8pBBAAEyV7FhlDG1h1qVnxBFgqbw== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBS/vhlhhJJDhAAC7VGdoH1RW2CD86QWRyLpA== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBV/vhlmoZaJRAAlJcQNSq2AU5Zjob73F3w/Q== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBY/vhlnqWyLxAAFHwkkmJ6zDfkWZSZrdIpPw== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBJ/vhl0ML6LRAArpEnyGv13LiAfPDPvZ9d/w== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBK/vhlW+kQJxAAroyc/q8pWnzRnFCDpbS6Gg== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBK/vhlOCW3LhAAsPJ8PI3iGi6FBWh3xMVT3g== 2024-03-19T02:54:36.228 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.f] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBK/vhlg1EbNhAAwYFwKUpXBMXaluHzVhJKcQ== 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBf/vhlXt4kBRAA5RiUaBfSS8kB43jLxY0PSQ== 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBi/vhlGfDgAxAAUfx6hzuUZhZfX3/V6MMbJw== 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBl/vhlLF1/ARAA65zetUBEo8s5x3PCQDAY4Q== 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBo/vhlPYwQABAA2JlIgwpjm10+RMr4f/2stw== 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-03-19T02:54:36.229 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBL/vhlp/38AxAANPDd6NA2hkWkcfVDWYlIsQ== 2024-03-19T02:54:36.287 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2024-03-19T02:54:36.287 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBJ/vhleUcgKRAAI4pxJ1Fs/nJrQjhSmo4arg== 2024-03-19T02:54:36.287 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBJ/vhl4j2pOBAAxAFwzNbUjM4AP5bCCFjHeg== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBK/vhl3shyDRAAOEJMRSRo49bo/kSEhvekKQ== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.e] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBK/vhlcUGPHRAAzpj7VtfXwV3nNS2G/fDRUQ== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBP/vhlCB8pBBAAEyV7FhlDG1h1qVnxBFgqbw== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBS/vhlhhJJDhAAC7VGdoH1RW2CD86QWRyLpA== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBV/vhlmoZaJRAAlJcQNSq2AU5Zjob73F3w/Q== 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2024-03-19T02:54:36.288 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBY/vhlnqWyLxAAFHwkkmJ6zDfkWZSZrdIpPw== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBJ/vhl0ML6LRAArpEnyGv13LiAfPDPvZ9d/w== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBK/vhlW+kQJxAAroyc/q8pWnzRnFCDpbS6Gg== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBK/vhlOCW3LhAAsPJ8PI3iGi6FBWh3xMVT3g== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.f] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBK/vhlg1EbNhAAwYFwKUpXBMXaluHzVhJKcQ== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBf/vhlXt4kBRAA5RiUaBfSS8kB43jLxY0PSQ== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBi/vhlGfDgAxAAUfx6hzuUZhZfX3/V6MMbJw== 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2024-03-19T02:54:36.289 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBl/vhlLF1/ARAA65zetUBEo8s5x3PCQDAY4Q== 2024-03-19T02:54:36.290 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2024-03-19T02:54:36.290 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBo/vhlPYwQABAA2JlIgwpjm10+RMr4f/2stw== 2024-03-19T02:54:36.290 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-03-19T02:54:36.290 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBL/vhlp/38AxAANPDd6NA2hkWkcfVDWYlIsQ== 2024-03-19T02:54:36.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T02:54:36.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T02:54:36.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.715 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:36.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:36.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:36.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:36.995 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.271 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T02:54:37.403 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T02:54:37.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T02:54:37.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:37.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.243 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T02:54:38.517 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T02:54:38.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T02:54:38.653 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-19T02:54:38.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-19T02:54:38.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T02:54:38.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-19T02:54:38.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-19T02:54:39.012 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T02:54:39.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-19T02:54:39.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-19T02:54:39.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T02:54:39.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-19T02:54:39.326 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:39.333 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T02:54:39.381 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-19T02:54:39.382 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-19T02:54:39.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-19T02:54:39.386 INFO:tasks.ceph.mon.a:Started 2024-03-19T02:54:39.387 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-19T02:54:39.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-19T02:54:39.424 INFO:tasks.ceph.mon.b:Started 2024-03-19T02:54:39.424 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-19T02:54:39.424 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-19T02:54:39.426 INFO:tasks.ceph.mon.c:Started 2024-03-19T02:54:39.426 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-19T02:54:39.426 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-19T02:54:39.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-19T02:54:39.431 INFO:tasks.ceph.mgr.y:Started 2024-03-19T02:54:39.431 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-19T02:54:39.431 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-19T02:54:39.433 INFO:tasks.ceph.mgr.x:Started 2024-03-19T02:54:39.433 INFO:tasks.ceph:Setting crush tunables to default 2024-03-19T02:54:39.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd crush tunables default 2024-03-19T02:54:39.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.608+0000 7f3842a00640 1 Processor -- start 2024-03-19T02:54:39.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.608+0000 7f3842a00640 1 -- start start 2024-03-19T02:54:39.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3842a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 0x7f383c15ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3842a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 0x7f383c0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3842a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f383c0ccbe0 con 0x7f383c154ec0 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3842a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f383c15f000 con 0x7f383c0cdcc0 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3842a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f383c1552a0 con 0x7f383c0cc230 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383be00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f383c154ec0 0x7f383c06cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54732/0 (socket says 172.21.15.92:54732) 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383be00640 1 -- 172.21.15.92:0/4082987002 learned_addr learned my addr 172.21.15.92:0/4082987002 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3840a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 0x7f383c0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 0x7f383c15ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3840a00640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 msgr2=0x7f383c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3840a00640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 msgr2=0x7f383c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3840a00640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 0x7f383c0cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:54:39.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f3840a00640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 0x7f383c0cc6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:54:39.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383b400640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 msgr2=0x7f383c15ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T02:54:39.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383b400640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 msgr2=0x7f383c15ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T02:54:39.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 0x7f383c15ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:54:39.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.610+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 0x7f383c15ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:54:39.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145458195 0 0) 0x7f383c0ccbe0 con 0x7f383c154ec0 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3820003520 con 0x7f383c154ec0 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1714333813 0 0) 0x7f3820003520 con 0x7f383c154ec0 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 msgr2=0x7f383c0cc6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.159:3300/0 conn(0x7f383c0cc230 0x7f383c0cc6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 msgr2=0x7f383c15ab50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 --2- 172.21.15.92:0/4082987002 >> v2:172.21.15.92:3300/0 conn(0x7f383c0cdcc0 0x7f383c15ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c15e480 con 0x7f383c154ec0 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7f382c001670 con 0x7f383c154ec0 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 >> v1:172.21.15.159:6789/0 conn(0x7f383c154ec0 legacy=0x7f383c06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:39.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3820003a00 0x7f3820003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f3820004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38200051c0 0x7f382000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15f000 con 0x7f3820003a00 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1552a0 con 0x7f38200051c0 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.621+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f382000c110 con 0x7f3820004800 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f3820004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383be00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3820003a00 0x7f3820003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 -- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38200051c0 msgr2=0x7f382000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f3840a00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38200051c0 0x7f382000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38200051c0 0x7f382000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 -- 172.21.15.92:0/4082987002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3820003a00 msgr2=0x7f3820003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3820003a00 0x7f3820003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383b400640 1 -- 172.21.15.92:0/4082987002 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3820003520 con 0x7f3820004800 2024-03-19T02:54:39.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f383be00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3820003a00 0x7f3820003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:54:39.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.622+0000 7f3840a00640 1 -- 172.21.15.92:0/4082987002 reap_dead start 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.623+0000 7f383b400640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f3820004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f382800b9e0 tx=0x7f3820005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a60277293fcc45e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.624+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382800b5e0 con 0x7f3820004800 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.624+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3828038070 con 0x7f3820004800 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.624+0000 7f383aa00640 1 -- 172.21.15.92:0/4082987002 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38280338b0 con 0x7f3820004800 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 -- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 msgr2=0x7f3820004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f3820004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f382800b9e0 tx=0x7f3820005830 comp rx=0 tx=0).stop 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 -- 172.21.15.92:0/4082987002 shutdown_connections 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 --2- 172.21.15.92:0/4082987002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f3820004c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 -- 172.21.15.92:0/4082987002 >> 172.21.15.92:0/4082987002 conn(0x7f383c06da80 msgr2=0x7f383c06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:39.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 -- 172.21.15.92:0/4082987002 shutdown_connections 2024-03-19T02:54:39.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.625+0000 7f3842a00640 1 -- 172.21.15.92:0/4082987002 wait complete. 2024-03-19T02:54:39.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.627+0000 7f3842a00640 1 Processor -- start 2024-03-19T02:54:39.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.627+0000 7f3842a00640 1 -- start start 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f383c15b840 0x7f383c1f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f383c0b4a60 0x7f383c1f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15ed20 con 0x7f383c15b840 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15ea40 con 0x7f383c0b4a60 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3842a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0cbd10 con 0x7f3820004800 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f383be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f383be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43502/0 (socket says 172.21.15.92:43502) 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f383b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f383c15b840 0x7f383c1f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f3840a00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f383c0b4a60 0x7f383c1f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:39.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.629+0000 7f383be00640 1 -- 172.21.15.92:0/1782976094 learned_addr learned my addr 172.21.15.92:0/1782976094 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 -- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f383c0b4a60 msgr2=0x7f383c1f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f383c0b4a60 0x7f383c1f14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 -- 172.21.15.92:0/1782976094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f383c15b840 msgr2=0x7f383c1f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f383c15b840 0x7f383c1f0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 -- 172.21.15.92:0/1782976094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c1f4230 con 0x7f3820004800 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.630+0000 7f383be00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d770 tx=0x7f382c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c019070 con 0x7f3820004800 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c0074e0 con 0x7f3820004800 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c1f44d0 con 0x7f3820004800 2024-03-19T02:54:39.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c004a00 con 0x7f3820004800 2024-03-19T02:54:39.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1f4a70 con 0x7f3820004800 2024-03-19T02:54:39.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f382c01c070 con 0x7f3820004800 2024-03-19T02:54:39.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.631+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f382c013470 con 0x7f3820004800 2024-03-19T02:54:39.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.632+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0684c0 con 0x7f3820004800 2024-03-19T02:54:39.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.636+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f382c023020 con 0x7f3820004800 2024-03-19T02:54:39.710 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.709+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f383c0c6680 con 0x7f3820004800 2024-03-19T02:54:39.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.715+0000 7f3833e00640 1 -- 172.21.15.92:0/1782976094 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f382c00b6b0 con 0x7f3820004800 2024-03-19T02:54:39.716 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to default 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 msgr2=0x7f383c0b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d770 tx=0x7f382c00dc30 comp rx=0 tx=0).stop 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 shutdown_connections 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f383c0b4a60 0x7f383c1f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f383c15b840 0x7f383c1f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 --2- 172.21.15.92:0/1782976094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3820004800 0x7f383c0b5170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.716+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 >> 172.21.15.92:0/1782976094 conn(0x7f383c06da80 msgr2=0x7f383c1537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.717+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 shutdown_connections 2024-03-19T02:54:39.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:39.717+0000 7f3842a00640 1 -- 172.21.15.92:0/1782976094 wait complete. 2024-03-19T02:54:39.734 INFO:tasks.ceph:check_enable_crimson: False 2024-03-19T02:54:39.734 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-19T02:54:39.734 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:39.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-19T02:54:39.801 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:39.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-19T02:54:39.924 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:39.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-19T02:54:40.052 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:54:40.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-19T02:54:40.054 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.054+0000 7f0995e38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:54:40.109 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.107+0000 7f0995e38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:54:40.183 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:40.183 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-19T02:54:40.196 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:40.195+0000 7f8b2722b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:54:40.209 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:40.209 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-19T02:54:40.258 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:40.257+0000 7f8b2722b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:54:40.276 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:40.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-19T02:54:40.343 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-19T02:54:40.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-19T02:54:40.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new e7ad93e3-f661-456e-953f-d55b62cefef0 0 2024-03-19T02:54:40.552 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.551+0000 7fbccb5d4640 1 Processor -- start 2024-03-19T02:54:40.552 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.551+0000 7fbccb5d4640 1 -- start start 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbccb5d4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbcc4151aa0 0x7fbcc40c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbccb5d4640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbcc40c9d60 0x7fbcc415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbccb5d4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcc4151e80 con 0x7fbcc40c8e10 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbccb5d4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbcc415e2d0 con 0x7fbcc4151aa0 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbccb5d4640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fbcc415ee50 con 0x7fbcc40c9d60 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbcc40c9d60 0x7fbcc415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbcc40c9d60 0x7fbcc415ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53274/0 (socket says 172.21.15.159:53274) 2024-03-19T02:54:40.554 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 -- 172.21.15.159:0/2088275693 learned_addr learned my addr 172.21.15.159:0/2088275693 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 -- 172.21.15.159:0/2088275693 >> v2:172.21.15.92:3300/0 conn(0x7fbcc4151aa0 msgr2=0x7fbcc40c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:40.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 --2- 172.21.15.159:0/2088275693 >> v2:172.21.15.92:3300/0 conn(0x7fbcc4151aa0 0x7fbcc40c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 -- 172.21.15.159:0/2088275693 >> v1:172.21.15.92:6789/0 conn(0x7fbcc40c8e10 legacy=0x7fbcc406c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:40.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.552+0000 7fbcc9b71640 1 -- 172.21.15.159:0/2088275693 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc40c9350 con 0x7fbcc40c9d60 2024-03-19T02:54:40.556 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbcc9b71640 1 --2- 172.21.15.159:0/2088275693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc415ace0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbcc406e240 tx=0x7fbcbc02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87bde45be8c16d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.556 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbcbbfff640 1 -- 172.21.15.159:0/2088275693 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc02ca80 con 0x7fbcc40c9d60 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbcbbfff640 1 -- 172.21.15.159:0/2088275693 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcbc02cc20 con 0x7fbcc40c9d60 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbcbbfff640 1 -- 172.21.15.159:0/2088275693 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc0343f0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 -- 172.21.15.159:0/2088275693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 msgr2=0x7fbcc415ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/2088275693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc415ace0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbcc406e240 tx=0x7fbcbc02bee0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 -- 172.21.15.159:0/2088275693 shutdown_connections 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/2088275693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc415ace0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/2088275693 >> v2:172.21.15.92:3300/0 conn(0x7fbcc4151aa0 0x7fbcc40c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.554+0000 7fbccb5d4640 1 -- 172.21.15.159:0/2088275693 >> 172.21.15.159:0/2088275693 conn(0x7fbcc4070420 msgr2=0x7fbcc406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.555+0000 7fbccb5d4640 1 -- 172.21.15.159:0/2088275693 shutdown_connections 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.555+0000 7fbccb5d4640 1 -- 172.21.15.159:0/2088275693 wait complete. 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 Processor -- start 2024-03-19T02:54:40.557 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 -- start start 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 0x7fbcc41f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcc4151aa0 0x7fbcc41e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc415eae0 con 0x7fbcc4151aa0 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc415e800 con 0x7fbcc40c9d60 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbccb5d4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc406ca40 con 0x7fbcc40c8e10 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbcc8b6f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbcc9370640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 0x7fbcc41f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbcc8b6f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53276/0 (socket says 172.21.15.159:53276) 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.556+0000 7fbcc8b6f640 1 -- 172.21.15.159:0/1233740802 learned_addr learned my addr 172.21.15.159:0/1233740802 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 -- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 msgr2=0x7fbcc41f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 0x7fbcc41f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 -- 172.21.15.159:0/1233740802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcc4151aa0 msgr2=0x7fbcc41e54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcc4151aa0 0x7fbcc41e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 -- 172.21.15.159:0/1233740802 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc4157d30 con 0x7fbcc40c9d60 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc9370640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 0x7fbcc41f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcc8b6f640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b410 tx=0x7fbcc000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0015020 con 0x7fbcc40c9d60 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc0011850 con 0x7fbcc40c9d60 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0011c40 con 0x7fbcc40c9d60 2024-03-19T02:54:40.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc4158010 con 0x7fbcc40c9d60 2024-03-19T02:54:40.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbcc0018020 con 0x7fbcc40c9d60 2024-03-19T02:54:40.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.557+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc4155770 con 0x7fbcc40c9d60 2024-03-19T02:54:40.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.558+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fbcc0011de0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.558+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc40686d0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.560+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbcc40686d0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.635+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e7ad93e3-f661-456e-953f-d55b62cefef0", "id": 0} v 0) -- 0x7fbcc406f9f0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.639+0000 7fbcb9ffb640 1 -- 172.21.15.159:0/1233740802 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7ad93e3-f661-456e-953f-d55b62cefef0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fbcc406f9f0 con 0x7fbcc40c9d60 2024-03-19T02:54:40.640 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 msgr2=0x7fbcc40c36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b410 tx=0x7fbcc000b810 comp rx=0 tx=0).stop 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 shutdown_connections 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcc4151aa0 0x7fbcc41e54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcc40c9d60 0x7fbcc40c36a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 --2- 172.21.15.159:0/1233740802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc40c8e10 0x7fbcc41f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 >> 172.21.15.159:0/1233740802 conn(0x7fbcc4070420 msgr2=0x7fbcc406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:40.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 shutdown_connections 2024-03-19T02:54:40.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.640+0000 7fbccb5d4640 1 -- 172.21.15.159:0/1233740802 wait complete. 2024-03-19T02:54:40.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new e60c183e-d354-46db-8324-3b1fd36c5c00 1 2024-03-19T02:54:40.659 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.659+0000 7f0995e38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:54:40.717 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.716+0000 7f0995e38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.751+0000 7ffa4ac81640 1 Processor -- start 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.751+0000 7ffa4ac81640 1 -- start start 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4ac81640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4ac81640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 0x7ffa441656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4ac81640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa4415f2c0 con 0x7ffa440c8fb0 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4ac81640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4406c350 con 0x7ffa440c53a0 2024-03-19T02:54:40.759 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4ac81640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa440cb2e0 con 0x7ffa4415eee0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa48a1d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4921e640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 0x7ffa441656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4921e640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 0x7ffa441656f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53286/0 (socket says 172.21.15.159:53286) 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa4921e640 1 -- 172.21.15.159:0/3609607978 learned_addr learned my addr 172.21.15.159:0/3609607978 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa48a1d640 1 -- 172.21.15.159:0/3609607978 >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 msgr2=0x7ffa441656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa48a1d640 1 --2- 172.21.15.159:0/3609607978 >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 0x7ffa441656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa48a1d640 1 -- 172.21.15.159:0/3609607978 >> v1:172.21.15.92:6789/0 conn(0x7ffa440c8fb0 legacy=0x7ffa4406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.752+0000 7ffa48a1d640 1 -- 172.21.15.159:0/3609607978 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa440cb5c0 con 0x7ffa4415eee0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.753+0000 7ffa48a1d640 1 --2- 172.21.15.159:0/3609607978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440ca920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa380097a0 tx=0x7ffa3802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da6039365d1d2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.753+0000 7ffa437fe640 1 -- 172.21.15.159:0/3609607978 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38037070 con 0x7ffa4415eee0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.753+0000 7ffa437fe640 1 -- 172.21.15.159:0/3609607978 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa38030710 con 0x7ffa4415eee0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.753+0000 7ffa437fe640 1 -- 172.21.15.159:0/3609607978 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38030ae0 con 0x7ffa4415eee0 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 -- 172.21.15.159:0/3609607978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 msgr2=0x7ffa440ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.760 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/3609607978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440ca920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa380097a0 tx=0x7ffa3802b710 comp rx=0 tx=0).stop 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 -- 172.21.15.159:0/3609607978 shutdown_connections 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/3609607978 >> v2:172.21.15.159:3300/0 conn(0x7ffa440c53a0 0x7ffa441656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/3609607978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440ca920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 -- 172.21.15.159:0/3609607978 >> 172.21.15.159:0/3609607978 conn(0x7ffa4406ff60 msgr2=0x7ffa44070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 -- 172.21.15.159:0/3609607978 shutdown_connections 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.754+0000 7ffa4ac81640 1 -- 172.21.15.159:0/3609607978 wait complete. 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.755+0000 7ffa4ac81640 1 Processor -- start 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.755+0000 7ffa4ac81640 1 -- start start 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa440c8fb0 0x7ffa440c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4406cb60 con 0x7ffa440c8fb0 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4406c880 con 0x7ffa440c53a0 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4ac81640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa440cb9e0 con 0x7ffa4415eee0 2024-03-19T02:54:40.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4921e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53300/0 (socket says 172.21.15.159:53300) 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa4921e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440c5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47072/0 (socket says 172.21.15.159:47072) 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 -- 172.21.15.159:0/2970791672 learned_addr learned my addr 172.21.15.159:0/2970791672 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 -- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 msgr2=0x7ffa440c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 -- 172.21.15.159:0/2970791672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa440c8fb0 msgr2=0x7ffa440c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa440c8fb0 0x7ffa440c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.756+0000 7ffa48a1d640 1 -- 172.21.15.159:0/2970791672 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa440b3820 con 0x7ffa440c53a0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa48a1d640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000c00 tx=0x7ffa380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38041020 con 0x7ffa440c53a0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3803c870 con 0x7ffa440c53a0 2024-03-19T02:54:40.762 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa441f0990 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38037040 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa441f0f30 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa38044020 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.757+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7ffa3803de00 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.758+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa440768a0 con 0x7ffa440c53a0 2024-03-19T02:54:40.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.761+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ffa3804b030 con 0x7ffa440c53a0 2024-03-19T02:54:40.767 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.766+0000 7f0995e38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:54:40.823 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.822+0000 7f0995e38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:54:40.835 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.834+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e60c183e-d354-46db-8324-3b1fd36c5c00", "id": 1} v 0) -- 0x7ffa440cbcf0 con 0x7ffa440c53a0 2024-03-19T02:54:40.840 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.839+0000 7ffa417fa640 1 -- 172.21.15.159:0/2970791672 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e60c183e-d354-46db-8324-3b1fd36c5c00", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7ffa3803cc60 con 0x7ffa440c53a0 2024-03-19T02:54:40.841 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 msgr2=0x7ffa440c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000c00 tx=0x7ffa380046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 shutdown_connections 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa4415eee0 0x7ffa440c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa440c8fb0 0x7ffa440c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 --2- 172.21.15.159:0/2970791672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa440c53a0 0x7ffa440c3840 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 >> 172.21.15.159:0/2970791672 conn(0x7ffa4406ff60 msgr2=0x7ffa440c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 shutdown_connections 2024-03-19T02:54:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.840+0000 7ffa4ac81640 1 -- 172.21.15.159:0/2970791672 wait complete. 2024-03-19T02:54:40.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 0f53fb99-a1cb-4f41-a48e-22949b3fadcb 2 2024-03-19T02:54:40.869 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.868+0000 7f0995e38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:54:40.880 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:40.879+0000 7f8b2722b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:54:40.944 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:40.943+0000 7f8b2722b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:54:40.970 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 Processor -- start 2024-03-19T02:54:40.970 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 -- start start 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff9901550b0 0x7ff99015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff990067ad0 0x7ff9900cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9900caf90 con 0x7ff990154cd0 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff99015e280 con 0x7ff9901550b0 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.969+0000 7ff9955b3640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff99015e520 con 0x7ff990067ad0 2024-03-19T02:54:40.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff990067ad0 0x7ff9900cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff990067ad0 0x7ff9900cbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53304/0 (socket says 172.21.15.159:53304) 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98effd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff990154cd0 0x7ff9900cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:39644/0 (socket says 172.21.15.159:39644) 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 -- 172.21.15.159:0/571909450 learned_addr learned my addr 172.21.15.159:0/571909450 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 -- 172.21.15.159:0/571909450 >> v2:172.21.15.92:3300/0 conn(0x7ff9901550b0 msgr2=0x7ff99015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 --2- 172.21.15.159:0/571909450 >> v2:172.21.15.92:3300/0 conn(0x7ff9901550b0 0x7ff99015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 -- 172.21.15.159:0/571909450 >> v1:172.21.15.159:6789/0 conn(0x7ff990154cd0 legacy=0x7ff9900cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98f7fe640 1 -- 172.21.15.159:0/571909450 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99015e800 con 0x7ff990067ad0 2024-03-19T02:54:40.972 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.970+0000 7ff98e7fc640 1 -- 172.21.15.159:0/571909450 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567010670 0 0) 0x7ff980001110 con 0x7ff990154cd0 2024-03-19T02:54:40.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff98f7fe640 1 --2- 172.21.15.159:0/571909450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990067ad0 0x7ff9900cbfa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff99006d760 tx=0x7ff98802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e394b9a96402e16d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff98e7fc640 1 -- 172.21.15.159:0/571909450 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98802cb20 con 0x7ff990067ad0 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff98e7fc640 1 -- 172.21.15.159:0/571909450 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff98802ccc0 con 0x7ff990067ad0 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff98e7fc640 1 -- 172.21.15.159:0/571909450 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff988034610 con 0x7ff990067ad0 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 -- 172.21.15.159:0/571909450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990067ad0 msgr2=0x7ff9900cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 --2- 172.21.15.159:0/571909450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990067ad0 0x7ff9900cbfa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff99006d760 tx=0x7ff98802bf80 comp rx=0 tx=0).stop 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 -- 172.21.15.159:0/571909450 shutdown_connections 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 --2- 172.21.15.159:0/571909450 >> v2:172.21.15.92:3300/0 conn(0x7ff9901550b0 0x7ff99015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 --2- 172.21.15.159:0/571909450 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990067ad0 0x7ff9900cbfa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 -- 172.21.15.159:0/571909450 >> 172.21.15.159:0/571909450 conn(0x7ff990072970 msgr2=0x7ff990066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 -- 172.21.15.159:0/571909450 shutdown_connections 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.971+0000 7ff9955b3640 1 -- 172.21.15.159:0/571909450 wait complete. 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 Processor -- start 2024-03-19T02:54:40.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 -- start start 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff990067ad0 0x7ff99015c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 0x7ff99014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff9900cd5c0 con 0x7ff990067ad0 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff99015edc0 con 0x7ff990154cd0 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff9955b3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff9900cd740 con 0x7ff9901550b0 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff987fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff987fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53312/0 (socket says 172.21.15.159:53312) 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff98f7fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 0x7ff99014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.973+0000 7ff987fff640 1 -- 172.21.15.159:0/3650727169 learned_addr learned my addr 172.21.15.159:0/3650727169 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 -- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 msgr2=0x7ff99014d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 0x7ff99014d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 -- 172.21.15.159:0/3650727169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff990067ad0 msgr2=0x7ff99015c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff990067ad0 0x7ff99015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 -- 172.21.15.159:0/3650727169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9900b55c0 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff98f7fe640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 0x7ff99014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff987fff640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b440 tx=0x7ff97800b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978015020 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff978011890 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9901f43f0 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978011c80 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9901f49b0 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff978018020 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.974+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ff978010ad0 con 0x7ff990154cd0 2024-03-19T02:54:40.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.975+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9900684c0 con 0x7ff990154cd0 2024-03-19T02:54:40.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:40.977+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff97801f020 con 0x7ff990154cd0 2024-03-19T02:54:40.993 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:40.992+0000 7f0995e38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:54:40.995 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:40.994+0000 7f8b2722b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:54:41.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.050+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0f53fb99-a1cb-4f41-a48e-22949b3fadcb", "id": 2} v 0) -- 0x7ff99014d170 con 0x7ff990154cd0 2024-03-19T02:54:41.057 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.056+0000 7ff9877fe640 1 -- 172.21.15.159:0/3650727169 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f53fb99-a1cb-4f41-a48e-22949b3fadcb", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ff97800d040 con 0x7ff990154cd0 2024-03-19T02:54:41.057 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.056+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 msgr2=0x7ff9900b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b440 tx=0x7ff97800b900 comp rx=0 tx=0).stop 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 shutdown_connections 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9901550b0 0x7ff99014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff990154cd0 0x7ff9900b5080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 --2- 172.21.15.159:0/3650727169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff990067ad0 0x7ff99015c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 >> 172.21.15.159:0/3650727169 conn(0x7ff990072970 msgr2=0x7ff99006c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 shutdown_connections 2024-03-19T02:54:41.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.057+0000 7ff9955b3640 1 -- 172.21.15.159:0/3650727169 wait complete. 2024-03-19T02:54:41.059 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.057+0000 7f8b2722b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:54:41.066 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.065+0000 7f0995e38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:54:41.068 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new b129c2e0-714e-48bc-96d4-307931384f8d 3 2024-03-19T02:54:41.107 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.106+0000 7f8b2722b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:54:41.116 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.115+0000 7f0995e38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:54:41.168 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.167+0000 7f62a36c6640 1 Processor -- start 2024-03-19T02:54:41.169 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 -- start start 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f629c151420 0x7f629c073500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629c0c7d50 con 0x7f629c15aa80 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f629c06feb0 con 0x7f629c151420 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.168+0000 7f62a36c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f629c06f0f0 con 0x7f629c0c8750 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f629c151420 0x7f629c073500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a0c61640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a0c61640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47102/0 (socket says 172.21.15.159:47102) 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f629c151420 0x7f629c073500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53316/0 (socket says 172.21.15.159:53316) 2024-03-19T02:54:41.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a0c61640 1 -- 172.21.15.159:0/473150564 learned_addr learned my addr 172.21.15.159:0/473150564 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 -- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 msgr2=0x7f629c15a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 -- 172.21.15.159:0/473150564 >> v1:172.21.15.92:6789/0 conn(0x7f629c15aa80 legacy=0x7f629c06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a1462640 1 -- 172.21.15.159:0/473150564 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c06f3d0 con 0x7f629c151420 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.169+0000 7f62a0c61640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f62a1462640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c151420 0x7f629c073500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62980097a0 tx=0x7f629802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc3bbbbfba5af7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f6293fff640 1 -- 172.21.15.159:0/473150564 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298035020 con 0x7f629c151420 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f6293fff640 1 -- 172.21.15.159:0/473150564 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6298031810 con 0x7f629c151420 2024-03-19T02:54:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f6293fff640 1 -- 172.21.15.159:0/473150564 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298031be0 con 0x7f629c151420 2024-03-19T02:54:41.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f62a36c6640 1 -- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c151420 msgr2=0x7f629c073500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.170+0000 7f62a36c6640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c151420 0x7f629c073500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62980097a0 tx=0x7f629802b8b0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 -- 172.21.15.159:0/473150564 shutdown_connections 2024-03-19T02:54:41.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c0c8750 0x7f629c15a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 --2- 172.21.15.159:0/473150564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c151420 0x7f629c073500 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 -- 172.21.15.159:0/473150564 >> 172.21.15.159:0/473150564 conn(0x7f629c0ccb50 msgr2=0x7f629c0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 -- 172.21.15.159:0/473150564 shutdown_connections 2024-03-19T02:54:41.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f62a36c6640 1 -- 172.21.15.159:0/473150564 wait complete. 2024-03-19T02:54:41.174 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.171+0000 7f0995e38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:54:41.176 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 Processor -- start 2024-03-19T02:54:41.176 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 -- start start 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 0x7f629c0cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f629c151420 0x7f629c1f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f629c0c7630 con 0x7f629c151420 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f629c06f990 con 0x7f629c0c8750 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.172+0000 7f62a36c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f629c1f3e30 con 0x7f629c15aa80 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1462640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 0x7f629c0cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47108/0 (socket says 172.21.15.159:47108) 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 -- 172.21.15.159:0/3869611933 learned_addr learned my addr 172.21.15.159:0/3869611933 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 -- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 msgr2=0x7f629c0cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 0x7f629c0cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 -- 172.21.15.159:0/3869611933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f629c151420 msgr2=0x7f629c1f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.177 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f629c151420 0x7f629c1f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 -- 172.21.15.159:0/3869611933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c154750 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1462640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 0x7f629c0cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a1c63640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f629400b9c0 tx=0x7f629400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294019070 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f629400cba0 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629c154a10 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294014440 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f629401c070 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.173+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f629c0c6460 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.174+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f629401dc00 con 0x7f629c15aa80 2024-03-19T02:54:41.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.174+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629c0c7240 con 0x7f629c15aa80 2024-03-19T02:54:41.179 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.176+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f629c0c7240 con 0x7f629c15aa80 2024-03-19T02:54:41.222 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.221+0000 7f0995e38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:54:41.237 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.236+0000 7f8b2722b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:54:41.250 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.249+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b129c2e0-714e-48bc-96d4-307931384f8d", "id": 3} v 0) -- 0x7f629c072010 con 0x7f629c15aa80 2024-03-19T02:54:41.256 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.255+0000 7f6291ffb640 1 -- 172.21.15.159:0/3869611933 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b129c2e0-714e-48bc-96d4-307931384f8d", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f629c072010 con 0x7f629c15aa80 2024-03-19T02:54:41.256 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 msgr2=0x7f629c1f4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f629400b9c0 tx=0x7f629400be80 comp rx=0 tx=0).stop 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 shutdown_connections 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f629c15aa80 0x7f629c1f4850 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f629c151420 0x7f629c1f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 --2- 172.21.15.159:0/3869611933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f629c0c8750 0x7f629c0cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.258 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 >> 172.21.15.159:0/3869611933 conn(0x7f629c0ccb50 msgr2=0x7f629c0cb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.258 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 shutdown_connections 2024-03-19T02:54:41.258 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.256+0000 7f62a36c6640 1 -- 172.21.15.159:0/3869611933 wait complete. 2024-03-19T02:54:41.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new d45a9953-14a3-47a4-9f53-6c7f7bac8b09 4 2024-03-19T02:54:41.316 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.315+0000 7f8b2722b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:54:41.366 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 Processor -- start 2024-03-19T02:54:41.367 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 -- start start 2024-03-19T02:54:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 0x7f92780c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92780708c0 con 0x7f92780c8ea0 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f92780c3a00 con 0x7f92780cb1a0 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.366+0000 7f927db77640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92780ca5d0 con 0x7f92780cb580 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f92777fe640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 0x7f92780c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f92777fe640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 0x7f92780c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53330/0 (socket says 172.21.15.159:53330) 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47116/0 (socket says 172.21.15.159:47116) 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f92777fe640 1 -- 172.21.15.159:0/1718622071 learned_addr learned my addr 172.21.15.159:0/1718622071 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 -- 172.21.15.159:0/1718622071 >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 msgr2=0x7f92780c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 --2- 172.21.15.159:0/1718622071 >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 0x7f92780c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 -- 172.21.15.159:0/1718622071 >> v1:172.21.15.92:6789/0 conn(0x7f92780c8ea0 legacy=0x7f92781527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.367+0000 7f9276ffd640 1 -- 172.21.15.159:0/1718622071 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92780cab00 con 0x7f92780cb580 2024-03-19T02:54:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f9276ffd640 1 --2- 172.21.15.159:0/1718622071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f926c0097a0 tx=0x7f926c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae6e25e2c948a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f92767fc640 1 -- 172.21.15.159:0/1718622071 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c035020 con 0x7f92780cb580 2024-03-19T02:54:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f92767fc640 1 -- 172.21.15.159:0/1718622071 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926c031830 con 0x7f92780cb580 2024-03-19T02:54:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f92767fc640 1 -- 172.21.15.159:0/1718622071 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c031c00 con 0x7f92780cb580 2024-03-19T02:54:41.370 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.368+0000 7f8b2722b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 -- 172.21.15.159:0/1718622071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 msgr2=0x7f9278158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 --2- 172.21.15.159:0/1718622071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f926c0097a0 tx=0x7f926c02b8d0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 -- 172.21.15.159:0/1718622071 shutdown_connections 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 --2- 172.21.15.159:0/1718622071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb580 0x7f9278158fc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 --2- 172.21.15.159:0/1718622071 >> v2:172.21.15.159:3300/0 conn(0x7f92780cb1a0 0x7f92780c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 -- 172.21.15.159:0/1718622071 >> 172.21.15.159:0/1718622071 conn(0x7f9278070e20 msgr2=0x7f9278071230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 -- 172.21.15.159:0/1718622071 shutdown_connections 2024-03-19T02:54:41.373 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.368+0000 7f927db77640 1 -- 172.21.15.159:0/1718622071 wait complete. 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.369+0000 7f927db77640 1 Processor -- start 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 -- start start 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92780c8ea0 0x7f927806c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92780cb580 0x7f92780cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9278158aa0 con 0x7f92780c8ea0 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f92780ca2f0 con 0x7f92780cb580 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f927db77640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92780ca7a0 con 0x7f92780cb1a0 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f9276ffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f9276ffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47124/0 (socket says 172.21.15.159:47124) 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f9276ffd640 1 -- 172.21.15.159:0/382766688 learned_addr learned my addr 172.21.15.159:0/382766688 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.370+0000 7f9277fff640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92780cb580 0x7f92780cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 -- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92780cb580 msgr2=0x7f92780cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.374 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92780cb580 0x7f92780cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 -- 172.21.15.159:0/382766688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92780c8ea0 msgr2=0x7f927806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92780c8ea0 0x7f927806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 -- 172.21.15.159:0/382766688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92780cc340 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f9276ffd640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f926c0098c0 tx=0x7f926c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c040070 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926c03e070 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92780cc620 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c035040 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92780ccbe0 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f926c038020 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.371+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f926c045ce0 con 0x7f92780cb1a0 2024-03-19T02:54:41.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.372+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92780768a0 con 0x7f92780cb1a0 2024-03-19T02:54:41.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.374+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f926c031800 con 0x7f92780cb1a0 2024-03-19T02:54:41.430 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.429+0000 7f8b2722b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:54:41.446 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.445+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d45a9953-14a3-47a4-9f53-6c7f7bac8b09", "id": 4} v 0) -- 0x7f92780c9560 con 0x7f92780cb1a0 2024-03-19T02:54:41.452 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.452+0000 7f925ffff640 1 -- 172.21.15.159:0/382766688 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d45a9953-14a3-47a4-9f53-6c7f7bac8b09", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f926c03ab70 con 0x7f92780cb1a0 2024-03-19T02:54:41.453 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 msgr2=0x7f92781e4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f926c0098c0 tx=0x7f926c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 shutdown_connections 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92780cb580 0x7f92780cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92780cb1a0 0x7f92781e4240 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 --2- 172.21.15.159:0/382766688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92780c8ea0 0x7f927806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 >> 172.21.15.159:0/382766688 conn(0x7f9278070e20 msgr2=0x7f9278151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 shutdown_connections 2024-03-19T02:54:41.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.453+0000 7f927db77640 1 -- 172.21.15.159:0/382766688 wait complete. 2024-03-19T02:54:41.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new aba312ec-c3d6-4903-9b99-37edcedc8ee9 5 2024-03-19T02:54:41.485 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.484+0000 7f8b2722b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:54:41.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.564+0000 7fa08cb5b640 1 Processor -- start 2024-03-19T02:54:41.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.565+0000 7fa08cb5b640 1 -- start start 2024-03-19T02:54:41.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08cb5b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa0880c7500 0x7fa088157b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08cb5b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 0x7fa088158090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08cb5b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08806fdf0 con 0x7fa088151a90 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08cb5b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa0880c7110 con 0x7fa0880c7500 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08cb5b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa08806fba0 con 0x7fa0880c7e10 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa086d9d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 0x7fa088158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08659c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa088151a90 0x7fa0880c6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:39660/0 (socket says 172.21.15.159:39660) 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08659c640 1 -- 172.21.15.159:0/1555139185 learned_addr learned my addr 172.21.15.159:0/1555139185 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa086d9d640 1 -- 172.21.15.159:0/1555139185 >> v2:172.21.15.92:3300/0 conn(0x7fa0880c7500 msgr2=0x7fa088157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa086d9d640 1 --2- 172.21.15.159:0/1555139185 >> v2:172.21.15.92:3300/0 conn(0x7fa0880c7500 0x7fa088157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa086d9d640 1 -- 172.21.15.159:0/1555139185 >> v1:172.21.15.159:6789/0 conn(0x7fa088151a90 legacy=0x7fa0880c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa08559a640 1 -- 172.21.15.159:0/1555139185 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552936663 0 0) 0x7fa08806fdf0 con 0x7fa088151a90 2024-03-19T02:54:41.568 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.566+0000 7fa086d9d640 1 -- 172.21.15.159:0/1555139185 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0880700d0 con 0x7fa0880c7e10 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.567+0000 7fa086d9d640 1 --2- 172.21.15.159:0/1555139185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 0x7fa088158090 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa080009d90 tx=0x7fa08002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8930c77f9e6c363 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.567+0000 7fa08559a640 1 -- 172.21.15.159:0/1555139185 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080039070 con 0x7fa0880c7e10 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.567+0000 7fa08559a640 1 -- 172.21.15.159:0/1555139185 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa08002cc50 con 0x7fa0880c7e10 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.567+0000 7fa08559a640 1 -- 172.21.15.159:0/1555139185 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080034440 con 0x7fa0880c7e10 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1555139185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 msgr2=0x7fa088158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1555139185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 0x7fa088158090 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa080009d90 tx=0x7fa08002bea0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1555139185 shutdown_connections 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1555139185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7e10 0x7fa088158090 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1555139185 >> v2:172.21.15.92:3300/0 conn(0x7fa0880c7500 0x7fa088157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1555139185 >> 172.21.15.159:0/1555139185 conn(0x7fa08806c350 msgr2=0x7fa08806c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1555139185 shutdown_connections 2024-03-19T02:54:41.570 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.568+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1555139185 wait complete. 2024-03-19T02:54:41.571 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.570+0000 7fa08cb5b640 1 Processor -- start 2024-03-19T02:54:41.571 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.570+0000 7fa08cb5b640 1 -- start start 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0880c7e10 0x7fa0881554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 0x7fa0881559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880c4f70 con 0x7fa0880c7e10 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880c4c90 con 0x7fa088151a90 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa088151e70 con 0x7fa0880c7500 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.574 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47148/0 (socket says 172.21.15.159:47148) 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa086d9d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 0x7fa0881559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 -- 172.21.15.159:0/1349452620 learned_addr learned my addr 172.21.15.159:0/1349452620 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 -- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 msgr2=0x7fa0881559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 0x7fa0881559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 -- 172.21.15.159:0/1349452620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0880c7e10 msgr2=0x7fa0881554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0880c7e10 0x7fa0881554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 -- 172.21.15.159:0/1349452620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0881f1410 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa086d9d640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 0x7fa0881559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08659c640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa07800c460 tx=0x7fa07800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078018070 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa078007500 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0881ed0e0 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078004a20 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.572+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0881ed680 con 0x7fa0880c7500 2024-03-19T02:54:41.575 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.573+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa07801b070 con 0x7fa0880c7500 2024-03-19T02:54:41.576 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.573+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa078012980 con 0x7fa0880c7500 2024-03-19T02:54:41.576 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.574+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0880768a0 con 0x7fa0880c7500 2024-03-19T02:54:41.577 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.576+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa078004bc0 con 0x7fa0880c7500 2024-03-19T02:54:41.627 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.626+0000 7f0995e38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:54:41.650 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.649+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aba312ec-c3d6-4903-9b99-37edcedc8ee9", "id": 5} v 0) -- 0x7fa08814ee40 con 0x7fa0880c7500 2024-03-19T02:54:41.658 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.657+0000 7fa076ffd640 1 -- 172.21.15.159:0/1349452620 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aba312ec-c3d6-4903-9b99-37edcedc8ee9", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa078012b80 con 0x7fa0880c7500 2024-03-19T02:54:41.658 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-03-19T02:54:41.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 msgr2=0x7fa088143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa07800c460 tx=0x7fa07800c920 comp rx=0 tx=0).stop 2024-03-19T02:54:41.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 shutdown_connections 2024-03-19T02:54:41.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa088151a90 0x7fa0881559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0880c7e10 0x7fa0881554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.660 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 --2- 172.21.15.159:0/1349452620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0880c7500 0x7fa088143790 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.660 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 >> 172.21.15.159:0/1349452620 conn(0x7fa08806c350 msgr2=0x7fa0880720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.660 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 shutdown_connections 2024-03-19T02:54:41.660 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.658+0000 7fa08cb5b640 1 -- 172.21.15.159:0/1349452620 wait complete. 2024-03-19T02:54:41.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new b3feeff9-5f92-4ca5-8d19-c1c46e993fca 6 2024-03-19T02:54:41.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.768+0000 7f696073b640 1 Processor -- start 2024-03-19T02:54:41.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.768+0000 7f696073b640 1 -- start start 2024-03-19T02:54:41.770 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.768+0000 7f0995e38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f696073b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69580c8fb0 0x7f695806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f696073b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f695815f2c0 con 0x7f695815eee0 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f696073b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69580cb2e0 con 0x7f69580c53a0 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f696073b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f695806c350 con 0x7f69580c8fb0 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f695ecd8640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f69580c53a0 0x7f69581656f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:39664/0 (socket says 172.21.15.159:39664) 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.769+0000 7f695ecd8640 1 -- 172.21.15.159:0/2504422731 learned_addr learned my addr 172.21.15.159:0/2504422731 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572987743 0 0) 0x7f69580cb2e0 con 0x7f69580c53a0 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6944003520 con 0x7f69580c53a0 2024-03-19T02:54:41.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1465117509 0 0) 0x7f6944003520 con 0x7f69580c53a0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 >> v2:172.21.15.92:3300/0 conn(0x7f69580c8fb0 msgr2=0x7f695806ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 --2- 172.21.15.159:0/2504422731 >> v2:172.21.15.92:3300/0 conn(0x7f69580c8fb0 0x7f695806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 >> v1:172.21.15.92:6789/0 conn(0x7f695815eee0 legacy=0x7f69580ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69580cb5c0 con 0x7f69580c53a0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7f69500019f0 con 0x7f69580c53a0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 >> v1:172.21.15.159:6789/0 conn(0x7f69580c53a0 legacy=0x7f69581656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f6944003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6944004800 0x7f6944004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69440053c0 0x7f694400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f695806c350 con 0x7f69440053c0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f694400c300 con 0x7f6944004800 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f694400c500 con 0x7f69440039b0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f6944003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695dcd6640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6944004800 0x7f6944004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695ecd8640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69440053c0 0x7f694400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 -- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6944004800 msgr2=0x7f6944004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6944004800 0x7f6944004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 -- 172.21.15.159:0/2504422731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69440053c0 msgr2=0x7f694400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69440053c0 0x7f694400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695e4d7640 1 -- 172.21.15.159:0/2504422731 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6944003520 con 0x7f69440039b0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695ecd8640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69440053c0 0x7f694400ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.770+0000 7f695ecd8640 1 -- 172.21.15.159:0/2504422731 reap_dead start 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.771+0000 7f695e4d7640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f6944003db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6948001650 tx=0x7f6944005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=246280546c24bce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.771+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948001220 con 0x7f69440039b0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.771+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6948005290 con 0x7f69440039b0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.771+0000 7f695d4d5640 1 -- 172.21.15.159:0/2504422731 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948003850 con 0x7f69440039b0 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 -- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 msgr2=0x7f6944003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.776 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f6944003db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6948001650 tx=0x7f6944005830 comp rx=0 tx=0).stop 2024-03-19T02:54:41.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 -- 172.21.15.159:0/2504422731 shutdown_connections 2024-03-19T02:54:41.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 --2- 172.21.15.159:0/2504422731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f6944003db0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 -- 172.21.15.159:0/2504422731 >> 172.21.15.159:0/2504422731 conn(0x7f695806ff60 msgr2=0x7f6958070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 -- 172.21.15.159:0/2504422731 shutdown_connections 2024-03-19T02:54:41.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.772+0000 7f696073b640 1 -- 172.21.15.159:0/2504422731 wait complete. 2024-03-19T02:54:41.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.774+0000 7f696073b640 1 Processor -- start 2024-03-19T02:54:41.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.774+0000 7f696073b640 1 -- start start 2024-03-19T02:54:41.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f695815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f695814e650 0x7f695814de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f695806cb60 con 0x7f695814e650 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f695806c880 con 0x7f69580b3720 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f696073b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f69580cb9e0 con 0x7f69440039b0 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f695ecd8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f695e4d7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f695815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f695ecd8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53352/0 (socket says 172.21.15.159:53352) 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f695ecd8640 1 -- 172.21.15.159:0/2653731878 learned_addr learned my addr 172.21.15.159:0/2653731878 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.775+0000 7f695e4d7640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f695815bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47152/0 (socket says 172.21.15.159:47152) 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 -- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 msgr2=0x7f695815bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f695815bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 -- 172.21.15.159:0/2653731878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f695814e650 msgr2=0x7f695814de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f695814e650 0x7f695814de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 -- 172.21.15.159:0/2653731878 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69580b1660 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f695ecd8640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69580ca2c0 tx=0x7f695000b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950004d70 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695000bb20 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950007070 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69580b1920 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69580b1ee0 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f695001b020 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.776+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6950014c50 con 0x7f69580b3720 2024-03-19T02:54:41.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.777+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69580b20e0 con 0x7f69580b3720 2024-03-19T02:54:41.782 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.779+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f69580b20e0 con 0x7f69580b3720 2024-03-19T02:54:41.852 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.851+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b3feeff9-5f92-4ca5-8d19-c1c46e993fca", "id": 6} v 0) -- 0x7f69581f4140 con 0x7f69580b3720 2024-03-19T02:54:41.859 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.858+0000 7f694effd640 1 -- 172.21.15.159:0/2653731878 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b3feeff9-5f92-4ca5-8d19-c1c46e993fca", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f6950022070 con 0x7f69580b3720 2024-03-19T02:54:41.859 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-03-19T02:54:41.860 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 msgr2=0x7f69581f4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.860 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69580ca2c0 tx=0x7f695000b4a0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.860 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 shutdown_connections 2024-03-19T02:54:41.860 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69580b3720 0x7f69581f4800 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.860 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f695814e650 0x7f695814de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.861 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 --2- 172.21.15.159:0/2653731878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69440039b0 0x7f695815bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.861 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 >> 172.21.15.159:0/2653731878 conn(0x7f695806ff60 msgr2=0x7f69580ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.861 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 shutdown_connections 2024-03-19T02:54:41.861 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.859+0000 7f696073b640 1 -- 172.21.15.159:0/2653731878 wait complete. 2024-03-19T02:54:41.870 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 676a4526-ecfc-4948-bfcd-d0b022bcfa9f 7 2024-03-19T02:54:41.902 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.901+0000 7f0995e38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:54:41.933 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:41.933+0000 7f8b2722b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:54:41.950 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:41.949+0000 7f0995e38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:54:41.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.970+0000 7f814780b640 1 Processor -- start 2024-03-19T02:54:41.971 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 -- start start 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 0x7f81400c35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f81401592e0 0x7f8140159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81400ca3c0 con 0x7f8140070e80 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f8140158e50 con 0x7f81401592e0 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.971+0000 7f814780b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f81400ca6a0 con 0x7f8140152e80 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f81455a7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 0x7f81400c35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f8145da8640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8140070e80 0x7f81400ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:39674/0 (socket says 172.21.15.159:39674) 2024-03-19T02:54:41.973 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f8145da8640 1 -- 172.21.15.159:0/3401360821 learned_addr learned my addr 172.21.15.159:0/3401360821 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f8137fff640 1 -- 172.21.15.159:0/3401360821 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219665928 0 0) 0x7f81400ca3c0 con 0x7f8140070e80 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f8137fff640 1 -- 172.21.15.159:0/3401360821 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f812c003520 con 0x7f8140070e80 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f81455a7640 1 -- 172.21.15.159:0/3401360821 >> v2:172.21.15.92:3300/0 conn(0x7f81401592e0 msgr2=0x7f8140159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f81455a7640 1 --2- 172.21.15.159:0/3401360821 >> v2:172.21.15.92:3300/0 conn(0x7f81401592e0 0x7f8140159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f81455a7640 1 -- 172.21.15.159:0/3401360821 >> v1:172.21.15.159:6789/0 conn(0x7f8140070e80 legacy=0x7f81400ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.974 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.972+0000 7f81455a7640 1 -- 172.21.15.159:0/3401360821 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81400ca980 con 0x7f8140152e80 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f81455a7640 1 --2- 172.21.15.159:0/3401360821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 0x7f81400c35d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f81300097a0 tx=0x7f813002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17dbc48bdb1ace98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f8137fff640 1 -- 172.21.15.159:0/3401360821 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130035020 con 0x7f8140152e80 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f8137fff640 1 -- 172.21.15.159:0/3401360821 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8130031810 con 0x7f8140152e80 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f8137fff640 1 -- 172.21.15.159:0/3401360821 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130031be0 con 0x7f8140152e80 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f814780b640 1 -- 172.21.15.159:0/3401360821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 msgr2=0x7f81400c35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.973+0000 7f814780b640 1 --2- 172.21.15.159:0/3401360821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 0x7f81400c35d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f81300097a0 tx=0x7f813002b8b0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 -- 172.21.15.159:0/3401360821 shutdown_connections 2024-03-19T02:54:41.976 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 --2- 172.21.15.159:0/3401360821 >> v2:172.21.15.92:3300/0 conn(0x7f81401592e0 0x7f8140159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.976 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 --2- 172.21.15.159:0/3401360821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8140152e80 0x7f81400c35d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.976 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 -- 172.21.15.159:0/3401360821 >> 172.21.15.159:0/3401360821 conn(0x7f814006fef0 msgr2=0x7f81400702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:41.976 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 -- 172.21.15.159:0/3401360821 shutdown_connections 2024-03-19T02:54:41.976 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.974+0000 7f814780b640 1 -- 172.21.15.159:0/3401360821 wait complete. 2024-03-19T02:54:41.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 Processor -- start 2024-03-19T02:54:41.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 -- start start 2024-03-19T02:54:41.978 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8140070e80 0x7f8140153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8140153260 con 0x7f8140070e80 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f81400cbf30 con 0x7f8140152e80 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f814780b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f81400c3b10 con 0x7f81401592e0 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f8144da6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.975+0000 7f8145da8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:53360/0 (socket says 172.21.15.159:53360) 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 -- 172.21.15.159:0/3263393047 learned_addr learned my addr 172.21.15.159:0/3263393047 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8145da8640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:47166/0 (socket says 172.21.15.159:47166) 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 -- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 msgr2=0x7f8140155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 -- 172.21.15.159:0/3263393047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8140070e80 msgr2=0x7f8140153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:41.979 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8140070e80 0x7f8140153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 -- 172.21.15.159:0/3263393047 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8140154bc0 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8145da8640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8144da6640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f813c00b440 tx=0x7f813c00b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c015020 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f813c011890 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8140154ea0 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c011c80 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f813c018020 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.976+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81400ce6f0 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.977+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f813c01a790 con 0x7f8140152e80 2024-03-19T02:54:41.980 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.977+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81400768a0 con 0x7f8140152e80 2024-03-19T02:54:41.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:41.979+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f813c01f020 con 0x7f8140152e80 2024-03-19T02:54:42.052 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.051+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "676a4526-ecfc-4948-bfcd-d0b022bcfa9f", "id": 7} v 0) -- 0x7f81400c9760 con 0x7f8140152e80 2024-03-19T02:54:42.058 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.058+0000 7f8135ffb640 1 -- 172.21.15.159:0/3263393047 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "676a4526-ecfc-4948-bfcd-d0b022bcfa9f", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f813c00d040 con 0x7f8140152e80 2024-03-19T02:54:42.059 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 msgr2=0x7f81400cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f813c00b440 tx=0x7f813c00b900 comp rx=0 tx=0).stop 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 shutdown_connections 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81401592e0 0x7f8140155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8140152e80 0x7f81400cec10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 --2- 172.21.15.159:0/3263393047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8140070e80 0x7f8140153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 >> 172.21.15.159:0/3263393047 conn(0x7f814006fef0 msgr2=0x7f814015bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:42.060 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 shutdown_connections 2024-03-19T02:54:42.061 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-19T02:54:42.059+0000 7f814780b640 1 -- 172.21.15.159:0/3263393047 wait complete. 2024-03-19T02:54:42.070 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-19T02:54:42.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-19T02:54:42.071 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:42.069+0000 7f0995e38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:54:42.078 INFO:tasks.ceph.osd.0:Started 2024-03-19T02:54:42.078 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-19T02:54:42.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-19T02:54:42.085 INFO:tasks.ceph.osd.1:Started 2024-03-19T02:54:42.085 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-19T02:54:42.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-19T02:54:42.091 INFO:tasks.ceph.osd.2:Started 2024-03-19T02:54:42.091 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-19T02:54:42.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-19T02:54:42.096 INFO:tasks.ceph.osd.3:Started 2024-03-19T02:54:42.096 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-19T02:54:42.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-19T02:54:42.096 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.095+0000 7f8b2722b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:54:42.113 INFO:tasks.ceph.osd.4:Started 2024-03-19T02:54:42.113 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-19T02:54:42.113 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-19T02:54:42.115 INFO:tasks.ceph.osd.5:Started 2024-03-19T02:54:42.115 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-19T02:54:42.115 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-19T02:54:42.117 INFO:tasks.ceph.osd.6:Started 2024-03-19T02:54:42.117 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-19T02:54:42.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-19T02:54:42.119 INFO:tasks.ceph.osd.7:Started 2024-03-19T02:54:42.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:42.197 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:42.196+0000 7f0995e38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:54:42.266 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.266+0000 7f8b2722b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:54:42.329 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.329+0000 7f8b2722b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:54:42.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.348+0000 7f4af7000640 1 Processor -- start 2024-03-19T02:54:42.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.348+0000 7f4af7000640 1 -- start start 2024-03-19T02:54:42.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af7000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 0x7f4af00c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:42.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af7000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4af015e320 0x7f4af0070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:42.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af7000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af00cb230 con 0x7f4af00c8e30 2024-03-19T02:54:42.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af7000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4af00ca4f0 con 0x7f4af01582a0 2024-03-19T02:54:42.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af7000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f4af00cace0 con 0x7f4af015e320 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4aefe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 0x7f4af00c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4aefe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 0x7f4af00c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53836/0 (socket says 172.21.15.92:53836) 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4ae7400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4af015e320 0x7f4af0070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4aefe00640 1 -- 172.21.15.92:0/2531237128 learned_addr learned my addr 172.21.15.92:0/2531237128 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.349+0000 7f4af5000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4af00c8e30 0x7f4af00ce380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:50046/0 (socket says 172.21.15.92:50046) 2024-03-19T02:54:42.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.350+0000 7f4ae7400640 1 -- 172.21.15.92:0/2531237128 >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 msgr2=0x7f4af00c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.350+0000 7f4ae7400640 1 --2- 172.21.15.92:0/2531237128 >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 0x7f4af00c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.350+0000 7f4ae7400640 1 -- 172.21.15.92:0/2531237128 >> v1:172.21.15.92:6789/0 conn(0x7f4af00c8e30 legacy=0x7f4af00ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.350+0000 7f4ae7400640 1 -- 172.21.15.92:0/2531237128 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af00c9f60 con 0x7f4af015e320 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.351+0000 7f4ae7400640 1 --2- 172.21.15.92:0/2531237128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af0070de0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4af006fc70 tx=0x7f4ad802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f80be129c95036 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.351+0000 7f4aef400640 1 -- 172.21.15.92:0/2531237128 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8035020 con 0x7f4af015e320 2024-03-19T02:54:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.351+0000 7f4aef400640 1 -- 172.21.15.92:0/2531237128 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad8031830 con 0x7f4af015e320 2024-03-19T02:54:42.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.351+0000 7f4aef400640 1 -- 172.21.15.92:0/2531237128 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8031c00 con 0x7f4af015e320 2024-03-19T02:54:42.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 -- 172.21.15.92:0/2531237128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 msgr2=0x7f4af0070de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 --2- 172.21.15.92:0/2531237128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af0070de0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4af006fc70 tx=0x7f4ad802b8d0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 -- 172.21.15.92:0/2531237128 shutdown_connections 2024-03-19T02:54:42.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 --2- 172.21.15.92:0/2531237128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af0070de0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 --2- 172.21.15.92:0/2531237128 >> v2:172.21.15.92:3300/0 conn(0x7f4af01582a0 0x7f4af00c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 -- 172.21.15.92:0/2531237128 >> 172.21.15.92:0/2531237128 conn(0x7f4af006f410 msgr2=0x7f4af006e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:42.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 -- 172.21.15.92:0/2531237128 shutdown_connections 2024-03-19T02:54:42.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.352+0000 7f4af7000640 1 -- 172.21.15.92:0/2531237128 wait complete. 2024-03-19T02:54:42.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.353+0000 7f4af7000640 1 Processor -- start 2024-03-19T02:54:42.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 -- start start 2024-03-19T02:54:42.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:42.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af01582a0 0x7f4af00c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:42.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af01ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:42.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00ca720 con 0x7f4af01582a0 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00cb7b0 con 0x7f4af015e320 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.354+0000 7f4af7000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4af00caa00 con 0x7f4af00c8e30 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4ae7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af01582a0 0x7f4af00c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4af5000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af01ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43546/0 (socket says 172.21.15.92:43546) 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 -- 172.21.15.92:0/2979965554 learned_addr learned my addr 172.21.15.92:0/2979965554 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 -- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 msgr2=0x7f4af01ee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af01ee5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 -- 172.21.15.92:0/2979965554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af01582a0 msgr2=0x7f4af00c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af01582a0 0x7f4af00c36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 -- 172.21.15.92:0/2979965554 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af00c4f20 con 0x7f4af00c8e30 2024-03-19T02:54:42.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.355+0000 7f4aefe00640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009d90 tx=0x7f4ae000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:42.357 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:42.355+0000 7f0995e38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:54:42.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae000cda0 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae0019070 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0014680 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af00c5200 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af00cd600 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ae0005d10 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.356+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4ae0014bb0 con 0x7f4af00c8e30 2024-03-19T02:54:42.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.357+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af00b47d0 con 0x7f4af00c8e30 2024-03-19T02:54:42.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.360+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4af00b47d0 con 0x7f4af00c8e30 2024-03-19T02:54:42.400 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:42.399+0000 7f0995e38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:54:42.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.423+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4af00684c0 con 0x7f4af00c8e30 2024-03-19T02:54:42.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.424+0000 7f4aecc00640 1 -- 172.21.15.92:0/2979965554 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4ae0010040 con 0x7f4af00c8e30 2024-03-19T02:54:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:42.424 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:42.054121+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 msgr2=0x7f4af01f1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009d90 tx=0x7f4ae000c490 comp rx=0 tx=0).stop 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 shutdown_connections 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af015e320 0x7f4af01ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af01582a0 0x7f4af00c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 --2- 172.21.15.92:0/2979965554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af00c8e30 0x7f4af01f1080 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 >> 172.21.15.92:0/2979965554 conn(0x7f4af006f410 msgr2=0x7f4af0153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.425+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 shutdown_connections 2024-03-19T02:54:42.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:42.426+0000 7f4af7000640 1 -- 172.21.15.92:0/2979965554 wait complete. 2024-03-19T02:54:42.441 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-19T02:54:42.442 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-19T02:54:42.463 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.463+0000 7f8b2722b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:54:42.555 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:42.554+0000 7f0995e38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:54:42.576 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.576+0000 7f8b2722b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:54:42.714 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-19T02:54:42.713+0000 7f88e4811640 -1 Falling back to public interface 2024-03-19T02:54:42.719 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-19T02:54:42.718+0000 7f4a3ed9a640 -1 Falling back to public interface 2024-03-19T02:54:42.724 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-19T02:54:42.723+0000 7f4358204640 -1 Falling back to public interface 2024-03-19T02:54:42.752 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-19T02:54:42.751+0000 7fab98e1a640 -1 Falling back to public interface 2024-03-19T02:54:42.753 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.752+0000 7f8b2722b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:54:42.800 INFO:tasks.ceph.osd.2.smithi092.stderr:2024-03-19T02:54:42.800+0000 7fd79020d640 -1 Falling back to public interface 2024-03-19T02:54:42.802 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.802+0000 7f8b2722b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:54:42.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-03-19T02:54:42.804+0000 7efc78c48640 -1 Falling back to public interface 2024-03-19T02:54:42.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2024-03-19T02:54:42.807+0000 7fe87f99a640 -1 Falling back to public interface 2024-03-19T02:54:42.810 INFO:tasks.ceph.osd.3.smithi092.stderr:2024-03-19T02:54:42.810+0000 7fa2cfc22640 -1 Falling back to public interface 2024-03-19T02:54:42.984 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:42.983+0000 7f8b2722b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:54:43.247 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:43.363 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:43.362+0000 7f0995e38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:54:43.472 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.471+0000 7f7879200640 1 Processor -- start 2024-03-19T02:54:43.472 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.472+0000 7f7879200640 1 -- start start 2024-03-19T02:54:43.473 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7879200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 0x7f78740c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:43.473 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7879200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78740c82a0 con 0x7f78740c86c0 2024-03-19T02:54:43.473 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7879200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f787406d280 con 0x7f7874071e50 2024-03-19T02:54:43.473 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7879200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f787406d560 con 0x7f7874152b50 2024-03-19T02:54:43.474 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7871400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f78740c86c0 0x7f7874158890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:50058/0 (socket says 172.21.15.92:50058) 2024-03-19T02:54:43.474 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7871400640 1 -- 172.21.15.92:0/275985179 learned_addr learned my addr 172.21.15.92:0/275985179 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:43.474 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.473+0000 7f7871e00640 1 --2- 172.21.15.92:0/275985179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 0x7f78740c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316229259 0 0) 0x7f787406d280 con 0x7f7874071e50 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7858003520 con 0x7f7874071e50 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2066216852 0 0) 0x7f78740c82a0 con 0x7f78740c86c0 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f787406d280 con 0x7f78740c86c0 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7871e00640 1 -- 172.21.15.92:0/275985179 >> v1:172.21.15.159:6789/0 conn(0x7f7874071e50 legacy=0x7f7874163210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7871e00640 1 -- 172.21.15.92:0/275985179 >> v1:172.21.15.92:6789/0 conn(0x7f78740c86c0 legacy=0x7f7874158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.475 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.474+0000 7f7871e00640 1 -- 172.21.15.92:0/275985179 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78740c5aa0 con 0x7f7874152b50 2024-03-19T02:54:43.476 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.475+0000 7f7871e00640 1 --2- 172.21.15.92:0/275985179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 0x7f78740c6bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78640097d0 tx=0x7f786402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83064bd518a23c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:43.476 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864035020 con 0x7f7874152b50 2024-03-19T02:54:43.476 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7864031890 con 0x7f7874152b50 2024-03-19T02:54:43.476 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7870a00640 1 -- 172.21.15.92:0/275985179 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864031c60 con 0x7f7874152b50 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 -- 172.21.15.92:0/275985179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 msgr2=0x7f78740c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 --2- 172.21.15.92:0/275985179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 0x7f78740c6bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78640097d0 tx=0x7f786402b930 comp rx=0 tx=0).stop 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 -- 172.21.15.92:0/275985179 shutdown_connections 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 --2- 172.21.15.92:0/275985179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874152b50 0x7f78740c6bf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 -- 172.21.15.92:0/275985179 >> 172.21.15.92:0/275985179 conn(0x7f78740cc1f0 msgr2=0x7f78740cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.476+0000 7f7879200640 1 -- 172.21.15.92:0/275985179 shutdown_connections 2024-03-19T02:54:43.477 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.477+0000 7f7879200640 1 -- 172.21.15.92:0/275985179 wait complete. 2024-03-19T02:54:43.478 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.478+0000 7f7879200640 1 Processor -- start 2024-03-19T02:54:43.478 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.478+0000 7f7879200640 1 -- start start 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78740c86c0 0x7f787414eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7874152b50 0x7f787415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7874158fa0 con 0x7f7874152b50 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7874152f30 con 0x7f78740c86c0 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7879200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f78740c62a0 con 0x7f7874071e50 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7871e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.479+0000 7f7871e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43560/0 (socket says 172.21.15.92:43560) 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 -- 172.21.15.92:0/1472760202 learned_addr learned my addr 172.21.15.92:0/1472760202 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871400640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78740c86c0 0x7f787414eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:43.481 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7872800640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7874152b50 0x7f787415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 -- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78740c86c0 msgr2=0x7f787414eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78740c86c0 0x7f787414eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 -- 172.21.15.92:0/1472760202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7874152b50 msgr2=0x7f787415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7874152b50 0x7f787415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 -- 172.21.15.92:0/1472760202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787414c450 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f7871e00640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7864034040 tx=0x7f78640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.480+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864040070 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7864030d20 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864035040 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787414c730 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f787414ae40 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f786404b060 con 0x7f7874071e50 2024-03-19T02:54:43.482 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.481+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7864044ad0 con 0x7f7874071e50 2024-03-19T02:54:43.483 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.482+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f787404f580 con 0x7f7874071e50 2024-03-19T02:54:43.484 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.484+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f786402d040 con 0x7f7874071e50 2024-03-19T02:54:43.551 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.551+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f787414cbd0 con 0x7f7874071e50 2024-03-19T02:54:43.552 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.552+0000 7f786a000640 1 -- 172.21.15.92:0/1472760202 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f786403a9f0 con 0x7f7874071e50 2024-03-19T02:54:43.552 INFO:teuthology.misc.health.smithi092.stdout: 2024-03-19T02:54:43.552 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":10,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:42.054121+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:43.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 msgr2=0x7f787414cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:43.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7864034040 tx=0x7f78640046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 shutdown_connections 2024-03-19T02:54:43.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7874152b50 0x7f787415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.554 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78740c86c0 0x7f787414eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.554 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 --2- 172.21.15.92:0/1472760202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7874071e50 0x7f787414cdf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:43.554 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 >> 172.21.15.92:0/1472760202 conn(0x7f78740cc1f0 msgr2=0x7f7874148e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:43.554 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 shutdown_connections 2024-03-19T02:54:43.554 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:43.553+0000 7f7879200640 1 -- 172.21.15.92:0/1472760202 wait complete. 2024-03-19T02:54:43.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-19T02:54:43.642 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:43.642+0000 7f0995e38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:54:43.694 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:43.693+0000 7f0995e38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:54:43.809 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:43.808+0000 7f0995e38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:54:44.018 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:44.018+0000 7f8b2722b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:54:44.330 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:44.330+0000 7f8b2722b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:54:44.389 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:44.389+0000 7f8b2722b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:54:44.522 INFO:tasks.ceph.mgr.y.smithi092.stderr:2024-03-19T02:54:44.522+0000 7f8b2722b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:54:44.847 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:44.846+0000 7f0936aa2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-19T02:54:44.847 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-19T02:54:44.846+0000 7f0936aa2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-19T02:54:46.081 INFO:tasks.ceph.osd.2.smithi092.stderr:2024-03-19T02:54:46.081+0000 7fd79020d640 -1 osd.2 0 log_to_monitors true 2024-03-19T02:54:46.178 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-19T02:54:46.177+0000 7f88e4811640 -1 osd.6 0 log_to_monitors true 2024-03-19T02:54:46.231 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-19T02:54:46.231+0000 7f4a3ed9a640 -1 osd.4 0 log_to_monitors true 2024-03-19T02:54:46.236 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-19T02:54:46.235+0000 7f4358204640 -1 osd.7 0 log_to_monitors true 2024-03-19T02:54:46.271 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-19T02:54:46.270+0000 7fab98e1a640 -1 osd.5 0 log_to_monitors true 2024-03-19T02:54:46.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-03-19T02:54:46.324+0000 7efc78c48640 -1 osd.1 0 log_to_monitors true 2024-03-19T02:54:46.369 INFO:tasks.ceph.osd.3.smithi092.stderr:2024-03-19T02:54:46.369+0000 7fa2cfc22640 -1 osd.3 0 log_to_monitors true 2024-03-19T02:54:46.432 INFO:tasks.ceph.osd.0.smithi092.stderr:2024-03-19T02:54:46.432+0000 7fe87f99a640 -1 osd.0 0 log_to_monitors true 2024-03-19T02:54:48.829 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-19T02:54:48.828+0000 7fab94761640 -1 osd.5 0 waiting for initial osdmap 2024-03-19T02:54:48.842 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-19T02:54:48.841+0000 7fab8c581640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:48.851 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-19T02:54:48.849+0000 7f435419a640 -1 osd.7 0 waiting for initial osdmap 2024-03-19T02:54:48.859 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-19T02:54:48.858+0000 7f4a3ad30640 -1 osd.4 0 waiting for initial osdmap 2024-03-19T02:54:48.865 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-19T02:54:48.864+0000 7f434bfba640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:48.866 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-19T02:54:48.866+0000 7f88e01af640 -1 osd.6 0 waiting for initial osdmap 2024-03-19T02:54:48.872 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-19T02:54:48.872+0000 7f4a32b50640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:48.877 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-19T02:54:48.876+0000 7f88d7fcf640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:49.266 INFO:tasks.ceph.osd.2.smithi092.stderr:2024-03-19T02:54:49.265+0000 7fd78aa00640 -1 osd.2 0 waiting for initial osdmap 2024-03-19T02:54:49.278 INFO:tasks.ceph.osd.2.smithi092.stderr:2024-03-19T02:54:49.277+0000 7fd780a00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:49.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-03-19T02:54:49.277+0000 7efc73a00640 -1 osd.1 0 waiting for initial osdmap 2024-03-19T02:54:49.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-03-19T02:54:49.286+0000 7efc69a00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:49.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2024-03-19T02:54:49.334+0000 7fa2ca600640 -1 osd.3 0 waiting for initial osdmap 2024-03-19T02:54:49.342 INFO:tasks.ceph.osd.3.smithi092.stderr:2024-03-19T02:54:49.341+0000 7fa2c0600640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:49.360 INFO:tasks.ceph.osd.0.smithi092.stderr:2024-03-19T02:54:49.359+0000 7fe87a800640 -1 osd.0 0 waiting for initial osdmap 2024-03-19T02:54:49.364 INFO:tasks.ceph.osd.0.smithi092.stderr:2024-03-19T02:54:49.363+0000 7fe870800640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:54:50.373 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:50.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.552+0000 7fcc6cc00640 1 Processor -- start 2024-03-19T02:54:50.553 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.552+0000 7fcc6cc00640 1 -- start start 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.553+0000 7fcc6cc00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc6806b4b0 0x7fcc6806bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.553+0000 7fcc6cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcc6806b890 0x7fcc680c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.553+0000 7fcc6cc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc6806e640 con 0x7fcc6806d070 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.553+0000 7fcc6cc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6806c350 con 0x7fcc6806b890 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.553+0000 7fcc6cc00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6815afa0 con 0x7fcc6806b4b0 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc6806b4b0 0x7fcc6806bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcc6806b4b0 0x7fcc6806bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:43156/0 (socket says 172.21.15.92:43156) 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 -- 172.21.15.92:0/1642520924 learned_addr learned my addr 172.21.15.92:0/1642520924 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:50.555 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65400640 1 --2- 172.21.15.92:0/1642520924 >> v2:172.21.15.92:3300/0 conn(0x7fcc6806b890 0x7fcc680c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.556 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 -- 172.21.15.92:0/1642520924 >> v2:172.21.15.92:3300/0 conn(0x7fcc6806b890 msgr2=0x7fcc680c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:50.556 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 --2- 172.21.15.92:0/1642520924 >> v2:172.21.15.92:3300/0 conn(0x7fcc6806b890 0x7fcc680c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.556 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 -- 172.21.15.92:0/1642520924 >> v1:172.21.15.92:6789/0 conn(0x7fcc6806d070 legacy=0x7fcc68154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.556 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.554+0000 7fcc65e00640 1 -- 172.21.15.92:0/1642520924 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6815b280 con 0x7fcc6806b4b0 2024-03-19T02:54:50.557 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.555+0000 7fcc65e00640 1 --2- 172.21.15.92:0/1642520924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b4b0 0x7fcc6806bd10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc68072e70 tx=0x7fcc5802c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74409e862bce2439 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:50.557 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc64a00640 1 -- 172.21.15.92:0/1642520924 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5802cba0 con 0x7fcc6806b4b0 2024-03-19T02:54:50.557 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc64a00640 1 -- 172.21.15.92:0/1642520924 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc5802cd40 con 0x7fcc6806b4b0 2024-03-19T02:54:50.557 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc64a00640 1 -- 172.21.15.92:0/1642520924 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58034610 con 0x7fcc6806b4b0 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1642520924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b4b0 msgr2=0x7fcc6806bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1642520924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b4b0 0x7fcc6806bd10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc68072e70 tx=0x7fcc5802c000 comp rx=0 tx=0).stop 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1642520924 shutdown_connections 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1642520924 >> v2:172.21.15.92:3300/0 conn(0x7fcc6806b890 0x7fcc680c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.556+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1642520924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b4b0 0x7fcc6806bd10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.557+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1642520924 >> 172.21.15.92:0/1642520924 conn(0x7fcc680cc1f0 msgr2=0x7fcc680cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.557+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1642520924 shutdown_connections 2024-03-19T02:54:50.558 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.557+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1642520924 wait complete. 2024-03-19T02:54:50.560 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.559+0000 7fcc6cc00640 1 Processor -- start 2024-03-19T02:54:50.560 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.559+0000 7fcc6cc00640 1 -- start start 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b890 0x7fcc681495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 0x7fcc6814e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6815ac70 con 0x7fcc6806b4b0 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6815b840 con 0x7fcc6806b890 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc6cc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcc680c7130 con 0x7fcc6806d070 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc65e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc65e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57872/0 (socket says 172.21.15.92:57872) 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc65e00640 1 -- 172.21.15.92:0/1156197294 learned_addr learned my addr 172.21.15.92:0/1156197294 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc66800640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 0x7fcc6814e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.560+0000 7fcc65400640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b890 0x7fcc681495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 -- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 msgr2=0x7fcc6814e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 0x7fcc6814e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.563 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 -- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b890 msgr2=0x7fcc681495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b890 0x7fcc681495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 -- 172.21.15.92:0/1156197294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc68149af0 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc66800640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 0x7fcc6814e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc65e00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009f40 tx=0x7fcc58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58043020 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc5803e450 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc68155a60 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.561+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58009510 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.562+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc68156020 con 0x7fcc6806b4b0 2024-03-19T02:54:50.564 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.563+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6804f580 con 0x7fcc6806b4b0 2024-03-19T02:54:50.565 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.564+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fcc58046020 con 0x7fcc6806b4b0 2024-03-19T02:54:50.566 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.565+0000 7fcc5e000640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 0x7fcc4806e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:50.566 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.565+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7fcc580ad0e0 con 0x7fcc6806b4b0 2024-03-19T02:54:50.569 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.566+0000 7fcc65400640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 0x7fcc4806e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:50.569 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.566+0000 7fcc65400640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 0x7fcc4806e030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc680c7520 tx=0x7fcc54009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:50.571 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.570+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcc5804d070 con 0x7fcc6806b4b0 2024-03-19T02:54:50.755 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.753+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc680c7930 con 0x7fcc6806b4b0 2024-03-19T02:54:50.756 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.755+0000 7fcc5e000640 1 -- 172.21.15.92:0/1156197294 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10909 (secure 0 0 0) 0x7fcc5807c160 con 0x7fcc6806b4b0 2024-03-19T02:54:50.756 INFO:teuthology.misc.health.smithi092.stdout: 2024-03-19T02:54:50.756 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":13,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:49.854498+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 msgr2=0x7fcc4806e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 0x7fcc4806e030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc680c7520 tx=0x7fcc54009190 comp rx=0 tx=0).stop 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 msgr2=0x7fcc681f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009f40 tx=0x7fcc58004a40 comp rx=0 tx=0).stop 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 shutdown_connections 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc4806bb70 0x7fcc4806e030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc6806d070 0x7fcc6814e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc6806b890 0x7fcc681495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.759 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 --2- 172.21.15.92:0/1156197294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc6806b4b0 0x7fcc681f0840 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:50.760 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.757+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 >> 172.21.15.92:0/1156197294 conn(0x7fcc680cc1f0 msgr2=0x7fcc680c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:50.760 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.758+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 shutdown_connections 2024-03-19T02:54:50.760 INFO:teuthology.misc.health.smithi092.stderr:2024-03-19T02:54:50.758+0000 7fcc6cc00640 1 -- 172.21.15.92:0/1156197294 wait complete. 2024-03-19T02:54:50.774 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-19T02:54:50.775 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-19T02:54:50.775 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-19T02:54:50.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-19T02:54:50.780 INFO:tasks.ceph.mds.a:Started 2024-03-19T02:54:50.780 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-19T02:54:50.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-19T02:54:50.786 INFO:tasks.ceph.mds.c:Started 2024-03-19T02:54:50.786 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-19T02:54:50.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-03-19T02:54:50.791 INFO:tasks.ceph.mds.e:Started 2024-03-19T02:54:50.791 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-19T02:54:50.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-19T02:54:50.794 INFO:tasks.ceph.mds.b:Started 2024-03-19T02:54:50.794 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-19T02:54:50.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-19T02:54:50.796 INFO:tasks.ceph.mds.d:Started 2024-03-19T02:54:50.796 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-19T02:54:50.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-19T02:54:50.798 INFO:tasks.ceph.mds.f:Started 2024-03-19T02:54:50.799 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-19T02:54:50.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:50.883 INFO:tasks.ceph.mds.f.smithi159.stdout:starting mds.f at 2024-03-19T02:54:50.884 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-03-19T02:54:50.884 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-19T02:54:50.930 INFO:tasks.ceph.mds.a.smithi092.stdout:starting mds.a at 2024-03-19T02:54:50.940 INFO:tasks.ceph.mds.e.smithi092.stdout:starting mds.e at 2024-03-19T02:54:50.951 INFO:tasks.ceph.mds.c.smithi092.stdout:starting mds.c at 2024-03-19T02:54:51.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:50.998+0000 7ff789400640 1 Processor -- start 2024-03-19T02:54:51.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:50.999+0000 7ff789400640 1 -- start start 2024-03-19T02:54:51.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff789400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff784071da0 0x7ff7840c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff789400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff784158db0 0x7ff784159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff789400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff784072890 con 0x7ff78406f800 2024-03-19T02:54:51.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff789400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff7840c6c90 con 0x7ff784158db0 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff789400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff7840c8200 con 0x7ff784071da0 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff783200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff784158db0 0x7ff784159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff77be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff78406f800 0x7ff784072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48478/0 (socket says 172.21.15.92:48478) 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff77be00640 1 -- 172.21.15.92:0/1937553821 learned_addr learned my addr 172.21.15.92:0/1937553821 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff782800640 1 --2- 172.21.15.92:0/1937553821 >> v2:172.21.15.159:3300/0 conn(0x7ff784071da0 0x7ff7840c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff783200640 1 -- 172.21.15.92:0/1937553821 >> v2:172.21.15.159:3300/0 conn(0x7ff784071da0 msgr2=0x7ff7840c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff783200640 1 --2- 172.21.15.92:0/1937553821 >> v2:172.21.15.159:3300/0 conn(0x7ff784071da0 0x7ff7840c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff783200640 1 -- 172.21.15.92:0/1937553821 >> v1:172.21.15.92:6789/0 conn(0x7ff78406f800 legacy=0x7ff784072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.000+0000 7ff781e00640 1 -- 172.21.15.92:0/1937553821 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730822113 0 0) 0x7ff784072890 con 0x7ff78406f800 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.001+0000 7ff783200640 1 -- 172.21.15.92:0/1937553821 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff784159740 con 0x7ff784158db0 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff783200640 1 --2- 172.21.15.92:0/1937553821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784158db0 0x7ff784159200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff784075650 tx=0x7ff77c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad6b6fee1b93625 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.007 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff781e00640 1 -- 172.21.15.92:0/1937553821 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c039070 con 0x7ff784158db0 2024-03-19T02:54:51.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff781e00640 1 -- 172.21.15.92:0/1937553821 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c02cc80 con 0x7ff784158db0 2024-03-19T02:54:51.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff781e00640 1 -- 172.21.15.92:0/1937553821 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c034660 con 0x7ff784158db0 2024-03-19T02:54:51.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff789400640 1 -- 172.21.15.92:0/1937553821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784158db0 msgr2=0x7ff784159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.002+0000 7ff789400640 1 --2- 172.21.15.92:0/1937553821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784158db0 0x7ff784159200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff784075650 tx=0x7ff77c02bed0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 -- 172.21.15.92:0/1937553821 shutdown_connections 2024-03-19T02:54:51.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 --2- 172.21.15.92:0/1937553821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784158db0 0x7ff784159200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 --2- 172.21.15.92:0/1937553821 >> v2:172.21.15.159:3300/0 conn(0x7ff784071da0 0x7ff7840c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 -- 172.21.15.92:0/1937553821 >> 172.21.15.92:0/1937553821 conn(0x7ff7840c8d90 msgr2=0x7ff7840c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:51.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 -- 172.21.15.92:0/1937553821 shutdown_connections 2024-03-19T02:54:51.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.003+0000 7ff789400640 1 -- 172.21.15.92:0/1937553821 wait complete. 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.004+0000 7ff789400640 1 Processor -- start 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.004+0000 7ff789400640 1 -- start start 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff78406f800 0x7ff78414d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 0x7ff7841f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff78415fde0 con 0x7ff784071da0 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff78415fb00 con 0x7ff784158db0 2024-03-19T02:54:51.015 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff789400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff7840c7120 con 0x7ff78406f800 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff77be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff77be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57962/0 (socket says 172.21.15.92:57962) 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.005+0000 7ff77be00640 1 -- 172.21.15.92:0/1975914252 learned_addr learned my addr 172.21.15.92:0/1975914252 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff782800640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff78406f800 0x7ff78414d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff783200640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 0x7ff7841f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 -- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff78406f800 msgr2=0x7ff78414d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff78406f800 0x7ff78414d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 -- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 msgr2=0x7ff7841f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 0x7ff7841f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 -- 172.21.15.92:0/1975914252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7840c5000 con 0x7ff784071da0 2024-03-19T02:54:51.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff783200640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 0x7ff7841f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77be00640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c4a0 tx=0x7ff76c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c018070 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76c0074e0 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7840c5270 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c004a20 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.006+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7840c5830 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.007+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78406df60 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.008+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7ff76c01b070 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.009+0000 7ff77aa00640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 0x7ff76006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.009+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7ff76c089870 con 0x7ff784071da0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.010+0000 7ff782800640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 0x7ff76006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.010+0000 7ff782800640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 0x7ff76006e000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff774009b40 tx=0x7ff774009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.014+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff78406df60 con 0x7ff784071da0 2024-03-19T02:54:51.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.199+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7840684c0 con 0x7ff784071da0 2024-03-19T02:54:51.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.201+0000 7ff77aa00640 1 -- 172.21.15.92:0/1975914252 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+12888 (secure 0 0 0) 0x7ff76c0580b0 con 0x7ff784071da0 2024-03-19T02:54:51.203 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:51.203 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:50.857665+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:51.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.203+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 msgr2=0x7ff76006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.203+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 0x7ff76006e000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff774009b40 tx=0x7ff774009190 comp rx=0 tx=0).stop 2024-03-19T02:54:51.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.203+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 msgr2=0x7ff7841f82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.203+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c4a0 tx=0x7ff76c00c960 comp rx=0 tx=0).stop 2024-03-19T02:54:51.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 shutdown_connections 2024-03-19T02:54:51.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff76006bb40 0x7ff76006e000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff784158db0 0x7ff7841f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff784071da0 0x7ff7841f82f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 --2- 172.21.15.92:0/1975914252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff78406f800 0x7ff78414d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 >> 172.21.15.92:0/1975914252 conn(0x7ff7840c8d90 msgr2=0x7ff78406e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:51.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 shutdown_connections 2024-03-19T02:54:51.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.204+0000 7ff789400640 1 -- 172.21.15.92:0/1975914252 wait complete. 2024-03-19T02:54:51.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:54:51.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:54:51.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.429+0000 7f1173800640 1 Processor -- start 2024-03-19T02:54:51.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.429+0000 7f1173800640 1 -- start start 2024-03-19T02:54:51.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.430+0000 7f1173800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.430+0000 7f1173800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f116c072380 0x7f116c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.430+0000 7f1173800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116c15b740 con 0x7f116c0c60d0 2024-03-19T02:54:51.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.430+0000 7f1173800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f116c15ba20 con 0x7f116c072380 2024-03-19T02:54:51.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.430+0000 7f1173800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f116c0c8400 con 0x7f116c154dc0 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46524/0 (socket says 172.21.15.92:46524) 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 -- 172.21.15.92:0/3933057865 learned_addr learned my addr 172.21.15.92:0/3933057865 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1171800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f116c072380 0x7f116c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 -- 172.21.15.92:0/3933057865 >> v2:172.21.15.159:3300/0 conn(0x7f116c072380 msgr2=0x7f116c0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 --2- 172.21.15.92:0/3933057865 >> v2:172.21.15.159:3300/0 conn(0x7f116c072380 0x7f116c0c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 -- 172.21.15.92:0/3933057865 >> v1:172.21.15.159:6789/0 conn(0x7f116c0c60d0 legacy=0x7f116c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.431+0000 7f1170e00640 1 -- 172.21.15.92:0/3933057865 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116c0c86e0 con 0x7f116c154dc0 2024-03-19T02:54:51.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.433+0000 7f1170e00640 1 --2- 172.21.15.92:0/3933057865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1160009870 tx=0x7f116002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a6b4c31cb0b13c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.433+0000 7f116b400640 1 -- 172.21.15.92:0/3933057865 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160035020 con 0x7f116c154dc0 2024-03-19T02:54:51.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.433+0000 7f116b400640 1 -- 172.21.15.92:0/3933057865 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1160031640 con 0x7f116c154dc0 2024-03-19T02:54:51.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.433+0000 7f116b400640 1 -- 172.21.15.92:0/3933057865 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116002d040 con 0x7f116c154dc0 2024-03-19T02:54:51.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 -- 172.21.15.92:0/3933057865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 msgr2=0x7f116c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 --2- 172.21.15.92:0/3933057865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1160009870 tx=0x7f116002b8f0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 -- 172.21.15.92:0/3933057865 shutdown_connections 2024-03-19T02:54:51.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 --2- 172.21.15.92:0/3933057865 >> v2:172.21.15.159:3300/0 conn(0x7f116c072380 0x7f116c0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 --2- 172.21.15.92:0/3933057865 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c071190 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 -- 172.21.15.92:0/3933057865 >> 172.21.15.92:0/3933057865 conn(0x7f116c0c8d90 msgr2=0x7f116c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:51.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 -- 172.21.15.92:0/3933057865 shutdown_connections 2024-03-19T02:54:51.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.434+0000 7f1173800640 1 -- 172.21.15.92:0/3933057865 wait complete. 2024-03-19T02:54:51.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.436+0000 7f1173800640 1 Processor -- start 2024-03-19T02:54:51.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.436+0000 7f1173800640 1 -- start start 2024-03-19T02:54:51.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f116c072380 0x7f116c1f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116c0c60d0 0x7f116c14dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f116c0c7350 con 0x7f116c0c60d0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f116c0c71d0 con 0x7f116c072380 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1173800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f116c0c74d0 con 0x7f116c154dc0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f116be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116c0c60d0 0x7f116c14dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46526/0 (socket says 172.21.15.92:46526) 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 -- 172.21.15.92:0/2636617360 learned_addr learned my addr 172.21.15.92:0/2636617360 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1170e00640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f116c072380 0x7f116c1f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 -- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f116c072380 msgr2=0x7f116c1f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f116c072380 0x7f116c1f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 -- 172.21.15.92:0/2636617360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116c0c60d0 msgr2=0x7f116c14dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116c0c60d0 0x7f116c14dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.437+0000 7f1171800640 1 -- 172.21.15.92:0/2636617360 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116c1f0b80 con 0x7f116c154dc0 2024-03-19T02:54:51.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1171800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f116400efc0 tx=0x7f116400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164018020 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1164014480 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164014870 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116c14d4d0 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.438+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f116c1f10d0 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.439+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f116c06cdf0 con 0x7f116c154dc0 2024-03-19T02:54:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.440+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f116401b020 con 0x7f116c154dc0 2024-03-19T02:54:51.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.441+0000 7f1168c00640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 0x7f114c06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.441+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7f116408b190 con 0x7f116c154dc0 2024-03-19T02:54:51.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.441+0000 7f1170e00640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 0x7f114c06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.442+0000 7f1170e00640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 0x7f114c06e000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1160036000 tx=0x7f116003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.446+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f116c06cdf0 con 0x7f116c154dc0 2024-03-19T02:54:51.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.626+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f116c1f12d0 con 0x7f116c154dc0 2024-03-19T02:54:51.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.627+0000 7f1168c00640 1 -- 172.21.15.92:0/2636617360 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v14) ==== 93+0+10 (secure 0 0 0) 0x7f116c1f12d0 con 0x7f116c154dc0 2024-03-19T02:54:51.628 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 msgr2=0x7f114c06e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 0x7f114c06e000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1160036000 tx=0x7f116003c040 comp rx=0 tx=0).stop 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 msgr2=0x7f116c1e5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f116400efc0 tx=0x7f116400c490 comp rx=0 tx=0).stop 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 shutdown_connections 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f114c06bb40 0x7f114c06e000 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116c154dc0 0x7f116c1e5b60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116c0c60d0 0x7f116c14dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 --2- 172.21.15.92:0/2636617360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f116c072380 0x7f116c1f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.629+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 >> 172.21.15.92:0/2636617360 conn(0x7f116c0c8d90 msgr2=0x7f116c0c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:51.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.630+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 shutdown_connections 2024-03-19T02:54:51.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.630+0000 7f1173800640 1 -- 172.21.15.92:0/2636617360 wait complete. 2024-03-19T02:54:51.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:54:51.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.838+0000 7f8406a00640 1 Processor -- start 2024-03-19T02:54:51.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.838+0000 7f8406a00640 1 -- start start 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8406a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8400152bf0 0x7f840006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8406a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f84001593d0 0x7f84000c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8406a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84000c9a80 con 0x7f84000c6470 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8406a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f84000c6c90 con 0x7f8400152bf0 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8406a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f8400159910 con 0x7f84001593d0 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f83ffe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8400152bf0 0x7f840006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f83ffe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8400152bf0 0x7f840006eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57990/0 (socket says 172.21.15.92:57990) 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f83ffe00640 1 -- 172.21.15.92:0/3447714170 learned_addr learned my addr 172.21.15.92:0/3447714170 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f83f7e00640 1 --1- 172.21.15.92:0/3447714170 >> v1:172.21.15.92:6789/0 conn(0x7f84000c6470 0x7f84000c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48488/0 (socket says 172.21.15.92:48488) 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.840+0000 7f8404a00640 1 --2- 172.21.15.92:0/3447714170 >> v2:172.21.15.159:3300/0 conn(0x7f84001593d0 0x7f84000c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.841+0000 7f83ffe00640 1 -- 172.21.15.92:0/3447714170 >> v2:172.21.15.159:3300/0 conn(0x7f84001593d0 msgr2=0x7f84000c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.841+0000 7f83ffe00640 1 --2- 172.21.15.92:0/3447714170 >> v2:172.21.15.159:3300/0 conn(0x7f84001593d0 0x7f84000c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.841+0000 7f83ffe00640 1 -- 172.21.15.92:0/3447714170 >> v1:172.21.15.92:6789/0 conn(0x7f84000c6470 legacy=0x7f84000c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.841+0000 7f83ffe00640 1 -- 172.21.15.92:0/3447714170 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000c8650 con 0x7f8400152bf0 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f83ffe00640 1 --2- 172.21.15.92:0/3447714170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8400152bf0 0x7f840006eef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84000c5070 tx=0x7f83f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6f26de5906dfb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f83ff400640 1 -- 172.21.15.92:0/3447714170 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0035020 con 0x7f8400152bf0 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f83ff400640 1 -- 172.21.15.92:0/3447714170 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f0031870 con 0x7f8400152bf0 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f83ff400640 1 -- 172.21.15.92:0/3447714170 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0031c40 con 0x7f8400152bf0 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f8406a00640 1 -- 172.21.15.92:0/3447714170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8400152bf0 msgr2=0x7f840006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f8406a00640 1 --2- 172.21.15.92:0/3447714170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8400152bf0 0x7f840006eef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84000c5070 tx=0x7f83f002b910 comp rx=0 tx=0).stop 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.842+0000 7f8406a00640 1 -- 172.21.15.92:0/3447714170 shutdown_connections 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.843+0000 7f8406a00640 1 --2- 172.21.15.92:0/3447714170 >> v2:172.21.15.159:3300/0 conn(0x7f84001593d0 0x7f84000c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.843+0000 7f8406a00640 1 --2- 172.21.15.92:0/3447714170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8400152bf0 0x7f840006eef0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.843+0000 7f8406a00640 1 -- 172.21.15.92:0/3447714170 >> 172.21.15.92:0/3447714170 conn(0x7f84000c8d90 msgr2=0x7f84000c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:51.846 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.843+0000 7f8406a00640 1 -- 172.21.15.92:0/3447714170 shutdown_connections 2024-03-19T02:54:51.846 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.843+0000 7f8406a00640 1 -- 172.21.15.92:0/3447714170 wait complete. 2024-03-19T02:54:51.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.844+0000 7f8406a00640 1 Processor -- start 2024-03-19T02:54:51.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.844+0000 7f8406a00640 1 -- start start 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 0x7f84001f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 0x7f8400072450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f840006ea60 con 0x7f84001593d0 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f840006e4a0 con 0x7f8400152bf0 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8406a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f840006e6f0 con 0x7f84000c6470 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f8404a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 0x7f8400072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f83ffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f83f7e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 0x7f84001f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f83ffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46532/0 (socket says 172.21.15.92:46532) 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.845+0000 7f83ffe00640 1 -- 172.21.15.92:0/2559731143 learned_addr learned my addr 172.21.15.92:0/2559731143 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 -- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 msgr2=0x7f84001f0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 0x7f84001f0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 -- 172.21.15.92:0/2559731143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 msgr2=0x7f8400072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 0x7f8400072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 -- 172.21.15.92:0/2559731143 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8400149a40 con 0x7f84000c6470 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f8404a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 0x7f8400072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83f7e00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 0x7f84001f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.846+0000 7f83ffe00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83f002b950 tx=0x7f83f00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.847+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0040070 con 0x7f84000c6470 2024-03-19T02:54:51.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.847+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f0030d00 con 0x7f84000c6470 2024-03-19T02:54:51.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.847+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0035040 con 0x7f84000c6470 2024-03-19T02:54:51.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.847+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8400149d20 con 0x7f84000c6470 2024-03-19T02:54:51.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.847+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001f0fe0 con 0x7f84000c6470 2024-03-19T02:54:51.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.848+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84000684c0 con 0x7f84000c6470 2024-03-19T02:54:51.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.849+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f83f004b060 con 0x7f84000c6470 2024-03-19T02:54:51.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.850+0000 7f83fcc00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 0x7f83dc06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:51.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.850+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7f83f00ac250 con 0x7f84000c6470 2024-03-19T02:54:51.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.850+0000 7f83f7e00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 0x7f83dc06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:51.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.851+0000 7f83f7e00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 0x7f83dc06e000 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84000c6ed0 tx=0x7f83e8007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:51.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:51.854+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f83f007ab20 con 0x7f84000c6470 2024-03-19T02:54:52.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.096+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840006f950 con 0x7f84000c6470 2024-03-19T02:54:52.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.097+0000 7f83fcc00640 1 -- 172.21.15.92:0/2559731143 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5521 (secure 0 0 0) 0x7f83f007ebc0 con 0x7f84000c6470 2024-03-19T02:54:52.099 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:52.099 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:54:52.099 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 3 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.099+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 msgr2=0x7f83dc06e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.099+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 0x7f83dc06e000 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84000c6ed0 tx=0x7f83e8007e80 comp rx=0 tx=0).stop 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.099+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 msgr2=0x7f8400072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.099+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83f002b950 tx=0x7f83f00046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 shutdown_connections 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f83dc06bb40 0x7f83dc06e000 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84001593d0 0x7f8400072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8400152bf0 0x7f84001f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 --2- 172.21.15.92:0/2559731143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84000c6470 0x7f8400072dd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 >> 172.21.15.92:0/2559731143 conn(0x7f84000c8d90 msgr2=0x7f840014ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:52.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 shutdown_connections 2024-03-19T02:54:52.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.100+0000 7f8406a00640 1 -- 172.21.15.92:0/2559731143 wait complete. 2024-03-19T02:54:52.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:52.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.307+0000 7f7e2f600640 1 Processor -- start 2024-03-19T02:54:52.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.307+0000 7f7e2f600640 1 -- start start 2024-03-19T02:54:52.310 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.308+0000 7f7e2f600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.310 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2f600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e280c3b80 con 0x7f7e280c7da0 2024-03-19T02:54:52.310 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2f600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e28072e70 con 0x7f7e28151ce0 2024-03-19T02:54:52.310 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2f600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7e28158860 con 0x7f7e28151900 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e1fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7e28151ce0 0x7f7e280723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:59912/0 (socket says 172.21.15.92:59912) 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46538/0 (socket says 172.21.15.92:46538) 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e1fe00640 1 -- 172.21.15.92:0/428881206 learned_addr learned my addr 172.21.15.92:0/428881206 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2cc00640 1 -- 172.21.15.92:0/428881206 >> v1:172.21.15.159:6789/0 conn(0x7f7e28151ce0 legacy=0x7f7e280723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2cc00640 1 -- 172.21.15.92:0/428881206 >> v1:172.21.15.92:6789/0 conn(0x7f7e280c7da0 legacy=0x7f7e28162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e2cc00640 1 -- 172.21.15.92:0/428881206 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e28073150 con 0x7f7e28151900 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.309+0000 7f7e27e00640 1 -- 172.21.15.92:0/428881206 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 662939158 0 0) 0x7f7e28072e70 con 0x7f7e28151ce0 2024-03-19T02:54:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.311+0000 7f7e2cc00640 1 --2- 172.21.15.92:0/428881206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097d0 tx=0x7f7e1802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2f9955813c858f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.311+0000 7f7e27e00640 1 -- 172.21.15.92:0/428881206 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18037070 con 0x7f7e28151900 2024-03-19T02:54:52.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.311+0000 7f7e27e00640 1 -- 172.21.15.92:0/428881206 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e18030710 con 0x7f7e28151900 2024-03-19T02:54:52.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.311+0000 7f7e27e00640 1 -- 172.21.15.92:0/428881206 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18030ae0 con 0x7f7e28151900 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 -- 172.21.15.92:0/428881206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 msgr2=0x7f7e28158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 --2- 172.21.15.92:0/428881206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097d0 tx=0x7f7e1802b800 comp rx=0 tx=0).stop 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 -- 172.21.15.92:0/428881206 shutdown_connections 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 --2- 172.21.15.92:0/428881206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151900 0x7f7e28158320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 -- 172.21.15.92:0/428881206 >> 172.21.15.92:0/428881206 conn(0x7f7e280c8d90 msgr2=0x7f7e280c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 -- 172.21.15.92:0/428881206 shutdown_connections 2024-03-19T02:54:52.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.312+0000 7f7e2f600640 1 -- 172.21.15.92:0/428881206 wait complete. 2024-03-19T02:54:52.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.313+0000 7f7e2f600640 1 Processor -- start 2024-03-19T02:54:52.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.314+0000 7f7e2f600640 1 -- start start 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.314+0000 7f7e2f600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.314+0000 7f7e2f600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e28151900 0x7f7e28154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2f600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151ce0 0x7f7e2806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2f600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280c9af0 con 0x7f7e280c7da0 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2f600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28071eb0 con 0x7f7e28151900 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2f600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7e280c9c70 con 0x7f7e28151ce0 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58016/0 (socket says 172.21.15.92:58016) 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 -- 172.21.15.92:0/91432377 learned_addr learned my addr 172.21.15.92:0/91432377 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:52.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2d600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151ce0 0x7f7e2806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e1fe00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e28151900 0x7f7e28154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 -- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151ce0 msgr2=0x7f7e2806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151ce0 0x7f7e2806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 -- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e28151900 msgr2=0x7f7e28154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e28151900 0x7f7e28154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 -- 172.21.15.92:0/91432377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281f1690 con 0x7f7e280c7da0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.315+0000 7f7e2cc00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e18035040 tx=0x7f7e180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.316+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18037040 con 0x7f7e280c7da0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.316+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e180329c0 con 0x7f7e280c7da0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.316+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e28152830 con 0x7f7e280c7da0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.316+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18032dd0 con 0x7f7e280c7da0 2024-03-19T02:54:52.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.316+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e2814f1e0 con 0x7f7e280c7da0 2024-03-19T02:54:52.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.317+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e280684c0 con 0x7f7e280c7da0 2024-03-19T02:54:52.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.318+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f7e18004030 con 0x7f7e280c7da0 2024-03-19T02:54:52.320 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.319+0000 7f7e25600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 0x7f7e0406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.320 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.319+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4857+0+0 (secure 0 0 0) 0x7f7e180ac460 con 0x7f7e280c7da0 2024-03-19T02:54:52.320 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.319+0000 7f7e1fe00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 0x7f7e0406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.320 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.320+0000 7f7e1fe00640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 0x7f7e0406e030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e10007f70 tx=0x7f7e10007f00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.324 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.323+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7e18042020 con 0x7f7e280c7da0 2024-03-19T02:54:52.524 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.523+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e280c4b80 con 0x7f7e280c7da0 2024-03-19T02:54:52.525 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.524+0000 7f7e25600640 1 -- 172.21.15.92:0/91432377 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12897 (secure 0 0 0) 0x7f7e1807ac00 con 0x7f7e280c7da0 2024-03-19T02:54:52.525 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:52.525 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":15,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:51.859723+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:52.528 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 msgr2=0x7f7e0406e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.528 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 0x7f7e0406e030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e10007f70 tx=0x7f7e10007f00 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 msgr2=0x7f7e2814f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e18035040 tx=0x7f7e180046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 shutdown_connections 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e0406bb70 0x7f7e0406e030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e28151ce0 0x7f7e2806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e28151900 0x7f7e28154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 --2- 172.21.15.92:0/91432377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e280c7da0 0x7f7e2814f700 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 >> 172.21.15.92:0/91432377 conn(0x7f7e280c8d90 msgr2=0x7f7e2814d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 shutdown_connections 2024-03-19T02:54:52.529 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.527+0000 7f7e2f600640 1 -- 172.21.15.92:0/91432377 wait complete. 2024-03-19T02:54:52.547 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:54:52.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:54:52.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.751+0000 7f92e3200640 1 Processor -- start 2024-03-19T02:54:52.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.751+0000 7f92e3200640 1 -- start start 2024-03-19T02:54:52.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e3200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 0x7f92e40c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e3200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e40c6d30 con 0x7f92e40c6460 2024-03-19T02:54:52.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e3200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e40c6ff0 con 0x7f92e41526a0 2024-03-19T02:54:52.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e3200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92e40c72d0 con 0x7f92e4152a80 2024-03-19T02:54:52.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 0x7f92e40c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e1e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f92e41526a0 0x7f92e4158800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:59922/0 (socket says 172.21.15.92:59922) 2024-03-19T02:54:52.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.752+0000 7f92e1e00640 1 -- 172.21.15.92:0/2272497487 learned_addr learned my addr 172.21.15.92:0/2272497487 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188875135 0 0) 0x7f92e40c6ff0 con 0x7f92e41526a0 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92c8003520 con 0x7f92e41526a0 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533466526 0 0) 0x7f92e40c6d30 con 0x7f92e40c6460 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92e40c6ff0 con 0x7f92e40c6460 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e1400640 1 -- 172.21.15.92:0/2272497487 >> v1:172.21.15.159:6789/0 conn(0x7f92e41526a0 legacy=0x7f92e4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e1400640 1 -- 172.21.15.92:0/2272497487 >> v1:172.21.15.92:6789/0 conn(0x7f92e40c6460 legacy=0x7f92e406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.753+0000 7f92e1400640 1 -- 172.21.15.92:0/2272497487 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e40c43d0 con 0x7f92e4152a80 2024-03-19T02:54:52.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.754+0000 7f92e1400640 1 --2- 172.21.15.92:0/2272497487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 0x7f92e40c3790 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f92d00097d0 tx=0x7f92d002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a0bb37e043dc5fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.754+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0034070 con 0x7f92e4152a80 2024-03-19T02:54:52.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.754+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d0030880 con 0x7f92e4152a80 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.754+0000 7f92e0a00640 1 -- 172.21.15.92:0/2272497487 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0030c50 con 0x7f92e4152a80 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 -- 172.21.15.92:0/2272497487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 msgr2=0x7f92e40c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 --2- 172.21.15.92:0/2272497487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 0x7f92e40c3790 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f92d00097d0 tx=0x7f92d002b9c0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 -- 172.21.15.92:0/2272497487 shutdown_connections 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 --2- 172.21.15.92:0/2272497487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e4152a80 0x7f92e40c3790 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 -- 172.21.15.92:0/2272497487 >> 172.21.15.92:0/2272497487 conn(0x7f92e406bac0 msgr2=0x7f92e40cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 -- 172.21.15.92:0/2272497487 shutdown_connections 2024-03-19T02:54:52.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.755+0000 7f92e3200640 1 -- 172.21.15.92:0/2272497487 wait complete. 2024-03-19T02:54:52.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.757+0000 7f92e3200640 1 Processor -- start 2024-03-19T02:54:52.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.757+0000 7f92e3200640 1 -- start start 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.757+0000 7f92e3200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e40c6460 0x7f92e41f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e3200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e41526a0 0x7f92e4151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92e40c3cd0 con 0x7f92e4152a80 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e3200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f92e40c4080 con 0x7f92e41526a0 2024-03-19T02:54:52.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e3200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92e40c8a50 con 0x7f92e40c6460 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58032/0 (socket says 172.21.15.92:58032) 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e1e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e40c6460 0x7f92e41f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 -- 172.21.15.92:0/3404239112 learned_addr learned my addr 172.21.15.92:0/3404239112 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e1400640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e41526a0 0x7f92e4151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 -- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e40c6460 msgr2=0x7f92e41f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e40c6460 0x7f92e41f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 -- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e41526a0 msgr2=0x7f92e4151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e41526a0 0x7f92e4151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.758+0000 7f92e2800640 1 -- 172.21.15.92:0/3404239112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e40cb0e0 con 0x7f92e4152a80 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92e2800640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c7c0 tx=0x7f92dc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92dc003b70 con 0x7f92e4152a80 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92dc003d10 con 0x7f92e4152a80 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92dc00aa50 con 0x7f92e4152a80 2024-03-19T02:54:52.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e40caca0 con 0x7f92e4152a80 2024-03-19T02:54:52.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.759+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e41f3cf0 con 0x7f92e4152a80 2024-03-19T02:54:52.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.760+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e406e140 con 0x7f92e4152a80 2024-03-19T02:54:52.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.761+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f92dc00abf0 con 0x7f92e4152a80 2024-03-19T02:54:52.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.762+0000 7f92da000640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 0x7f92c406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:52.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.762+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4857+0+0 (secure 0 0 0) 0x7f92dc08a530 con 0x7f92e4152a80 2024-03-19T02:54:52.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.762+0000 7f92e1e00640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 0x7f92c406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:52.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.763+0000 7f92e1e00640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 0x7f92c406e030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f92d4000f60 tx=0x7f92d4007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:52.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.768+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f92e406e140 con 0x7f92e4152a80 2024-03-19T02:54:52.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.972+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92e404f580 con 0x7f92e4152a80 2024-03-19T02:54:52.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.972+0000 7f92da000640 1 -- 172.21.15.92:0/3404239112 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f92dc058cd0 con 0x7f92e4152a80 2024-03-19T02:54:52.974 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:54:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 msgr2=0x7f92c406e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 0x7f92c406e030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f92d4000f60 tx=0x7f92d4007d30 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 msgr2=0x7f92e41e4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c7c0 tx=0x7f92dc00cc80 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 shutdown_connections 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f92c406bb70 0x7f92c406e030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92e4152a80 0x7f92e41e4a40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92e41526a0 0x7f92e4151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 --2- 172.21.15.92:0/3404239112 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92e40c6460 0x7f92e41f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 >> 172.21.15.92:0/3404239112 conn(0x7f92e406bac0 msgr2=0x7f92e414d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 shutdown_connections 2024-03-19T02:54:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:52.975+0000 7f92e3200640 1 -- 172.21.15.92:0/3404239112 wait complete. 2024-03-19T02:54:52.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T02:54:53.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.180+0000 7fceabe00640 1 Processor -- start 2024-03-19T02:54:53.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.181+0000 7fceabe00640 1 -- start start 2024-03-19T02:54:53.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fceabe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 0x7fcea415e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fceabe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea40ca8e0 con 0x7fcea40c8e30 2024-03-19T02:54:53.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fceabe00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea40caba0 con 0x7fcea40c52c0 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fceabe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcea40c96f0 con 0x7fcea40b0d70 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fcea8a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fcea40c8e30 0x7fcea4070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48518/0 (socket says 172.21.15.92:48518) 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fcea8a00640 1 -- 172.21.15.92:0/2421598798 learned_addr learned my addr 172.21.15.92:0/2421598798 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.182+0000 7fcea9400640 1 --2- 172.21.15.92:0/2421598798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 0x7fcea415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.183+0000 7fcea3e00640 1 -- 172.21.15.92:0/2421598798 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26117291 0 0) 0x7fcea40caba0 con 0x7fcea40c52c0 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.183+0000 7fcea9400640 1 -- 172.21.15.92:0/2421598798 >> v1:172.21.15.159:6789/0 conn(0x7fcea40c52c0 legacy=0x7fcea4165650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.183+0000 7fcea9400640 1 -- 172.21.15.92:0/2421598798 >> v1:172.21.15.92:6789/0 conn(0x7fcea40c8e30 legacy=0x7fcea4070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.183+0000 7fcea9400640 1 -- 172.21.15.92:0/2421598798 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea40cafe0 con 0x7fcea40b0d70 2024-03-19T02:54:53.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.184+0000 7fcea9400640 1 --2- 172.21.15.92:0/2421598798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 0x7fcea415e320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c9870 tx=0x7fce9802b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a9548d0336a2b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.184+0000 7fcea3e00640 1 -- 172.21.15.92:0/2421598798 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98037070 con 0x7fcea40b0d70 2024-03-19T02:54:53.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.184+0000 7fcea3e00640 1 -- 172.21.15.92:0/2421598798 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce98030710 con 0x7fcea40b0d70 2024-03-19T02:54:53.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.184+0000 7fcea3e00640 1 -- 172.21.15.92:0/2421598798 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98030ae0 con 0x7fcea40b0d70 2024-03-19T02:54:53.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 -- 172.21.15.92:0/2421598798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 msgr2=0x7fcea415e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 --2- 172.21.15.92:0/2421598798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 0x7fcea415e320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c9870 tx=0x7fce9802b760 comp rx=0 tx=0).stop 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 -- 172.21.15.92:0/2421598798 shutdown_connections 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 --2- 172.21.15.92:0/2421598798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40b0d70 0x7fcea415e320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 -- 172.21.15.92:0/2421598798 >> 172.21.15.92:0/2421598798 conn(0x7fcea406f410 msgr2=0x7fcea406e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 -- 172.21.15.92:0/2421598798 shutdown_connections 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.185+0000 7fceabe00640 1 -- 172.21.15.92:0/2421598798 wait complete. 2024-03-19T02:54:53.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.186+0000 7fceabe00640 1 Processor -- start 2024-03-19T02:54:53.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.187+0000 7fceabe00640 1 -- start start 2024-03-19T02:54:53.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.187+0000 7fceabe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.187+0000 7fceabe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcea40c52c0 0x7fcea41ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.187+0000 7fceabe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40c8e30 0x7fcea41534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fceabe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcea40cb880 con 0x7fcea40b0d70 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fceabe00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcea40cb5a0 con 0x7fcea40c52c0 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fceabe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcea40ca4f0 con 0x7fcea40c8e30 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58042/0 (socket says 172.21.15.92:58042) 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 -- 172.21.15.92:0/3175450872 learned_addr learned my addr 172.21.15.92:0/3175450872 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea8a00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcea40c52c0 0x7fcea41ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9e00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40c8e30 0x7fcea41534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 -- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40c8e30 msgr2=0x7fcea41534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40c8e30 0x7fcea41534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 -- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcea40c52c0 msgr2=0x7fcea41ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcea40c52c0 0x7fcea41ee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.188+0000 7fcea9400640 1 -- 172.21.15.92:0/3175450872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea415baa0 con 0x7fcea40b0d70 2024-03-19T02:54:53.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fcea9400640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce980098c0 tx=0x7fce980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98042070 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce98032a40 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98037040 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea415a370 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.189+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea415a930 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.190+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea40684c0 con 0x7fcea40b0d70 2024-03-19T02:54:53.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.191+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fce98045070 con 0x7fcea40b0d70 2024-03-19T02:54:53.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.192+0000 7fcea1600640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 0x7fce8406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.192+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fce980ac3b0 con 0x7fcea40b0d70 2024-03-19T02:54:53.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.192+0000 7fcea8a00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 0x7fce8406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.193+0000 7fcea8a00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 0x7fce8406e030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcea4071630 tx=0x7fce90007e20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.196+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fce9807ad00 con 0x7fcea40b0d70 2024-03-19T02:54:53.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.397+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcea40b3990 con 0x7fcea40b0d70 2024-03-19T02:54:53.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.398+0000 7fcea1600640 1 -- 172.21.15.92:0/3175450872 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7fce9807eda0 con 0x7fcea40b0d70 2024-03-19T02:54:53.399 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:53.399 INFO:teuthology.orchestra.run.smithi092.stdout:[ 2024-03-19T02:54:53.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-19T02:54:53.399 INFO:teuthology.orchestra.run.smithi092.stdout: "poolnum": 1, 2024-03-19T02:54:53.400 INFO:teuthology.orchestra.run.smithi092.stdout: "poolname": ".mgr" 2024-03-19T02:54:53.400 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-19T02:54:53.400 INFO:teuthology.orchestra.run.smithi092.stdout:] 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 msgr2=0x7fce8406e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 0x7fce8406e030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcea4071630 tx=0x7fce90007e20 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 msgr2=0x7fcea4158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce980098c0 tx=0x7fce980046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 shutdown_connections 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fce8406bb70 0x7fce8406e030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcea40c8e30 0x7fcea41534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcea40c52c0 0x7fcea41ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 --2- 172.21.15.92:0/3175450872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcea40b0d70 0x7fcea4158c20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 >> 172.21.15.92:0/3175450872 conn(0x7fcea406f410 msgr2=0x7fcea40af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 shutdown_connections 2024-03-19T02:54:53.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.400+0000 7fceabe00640 1 -- 172.21.15.92:0/3175450872 wait complete. 2024-03-19T02:54:53.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T02:54:53.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T02:54:53.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.605+0000 7f3d0ea00640 1 Processor -- start 2024-03-19T02:54:53.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.605+0000 7f3d0ea00640 1 -- start start 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d080c81a0 0x7f3d0815b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 0x7f3d081657e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ea00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d080c9250 con 0x7f3d080c7dc0 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ea00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3d0815b1e0 con 0x7f3d080c81a0 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3d0815e890 con 0x7f3d0815bbb0 2024-03-19T02:54:53.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d07400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d080c81a0 0x7f3d0815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d07400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d080c81a0 0x7f3d0815b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58058/0 (socket says 172.21.15.92:58058) 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d07400640 1 -- 172.21.15.92:0/1263950553 learned_addr learned my addr 172.21.15.92:0/1263950553 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d07e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3d080c7dc0 0x7f3d080c8b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48520/0 (socket says 172.21.15.92:48520) 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.606+0000 7f3d0ca00640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 0x7f3d081657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d06a00640 1 -- 172.21.15.92:0/1263950553 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836442618 0 0) 0x7f3d080c9250 con 0x7f3d080c7dc0 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d06a00640 1 -- 172.21.15.92:0/1263950553 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cec003520 con 0x7f3d080c7dc0 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d07400640 1 -- 172.21.15.92:0/1263950553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 msgr2=0x7f3d081657e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d07400640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 0x7f3d081657e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d07400640 1 -- 172.21.15.92:0/1263950553 >> v1:172.21.15.92:6789/0 conn(0x7f3d080c7dc0 legacy=0x7f3d080c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d07400640 1 -- 172.21.15.92:0/1263950553 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0815eb70 con 0x7f3d080c81a0 2024-03-19T02:54:53.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.607+0000 7f3d0ca00640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 0x7f3d081657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:53.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.608+0000 7f3d07400640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d080c81a0 0x7f3d0815b670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009870 tx=0x7f3cf402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac3e3e83a409a49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.608+0000 7f3d06a00640 1 -- 172.21.15.92:0/1263950553 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf402f830 con 0x7f3d080c81a0 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.608+0000 7f3d06a00640 1 -- 172.21.15.92:0/1263950553 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf402f9d0 con 0x7f3d080c81a0 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.608+0000 7f3d06a00640 1 -- 172.21.15.92:0/1263950553 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf402fd80 con 0x7f3d080c81a0 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1263950553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d080c81a0 msgr2=0x7f3d0815b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d080c81a0 0x7f3d0815b670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009870 tx=0x7f3cf402ba10 comp rx=0 tx=0).stop 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1263950553 shutdown_connections 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d0815bbb0 0x7f3d081657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1263950553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d080c81a0 0x7f3d0815b670 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1263950553 >> 172.21.15.92:0/1263950553 conn(0x7f3d080cd2a0 msgr2=0x7f3d080cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1263950553 shutdown_connections 2024-03-19T02:54:53.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.609+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1263950553 wait complete. 2024-03-19T02:54:53.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.611+0000 7f3d0ea00640 1 Processor -- start 2024-03-19T02:54:53.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.611+0000 7f3d0ea00640 1 -- start start 2024-03-19T02:54:53.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080c7dc0 0x7f3d08151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080c81a0 0x7f3d0814cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3d080c85c0 con 0x7f3d0815bbb0 2024-03-19T02:54:53.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0815f130 con 0x7f3d080c81a0 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3d08155640 con 0x7f3d080c7dc0 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d07400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080c81a0 0x7f3d0814cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080c7dc0 0x7f3d08151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58068/0 (socket says 172.21.15.92:58068) 2024-03-19T02:54:53.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 -- 172.21.15.92:0/1908366188 learned_addr learned my addr 172.21.15.92:0/1908366188 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 -- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080c7dc0 msgr2=0x7f3d08151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080c7dc0 0x7f3d08151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 -- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080c81a0 msgr2=0x7f3d0814cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080c81a0 0x7f3d0814cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.612+0000 7f3d0ca00640 1 -- 172.21.15.92:0/1908366188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d081f1a00 con 0x7f3d0815bbb0 2024-03-19T02:54:53.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3d0ca00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e960 tx=0x7f3d0000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00017070 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d00004020 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d000043f0 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d081f1ce0 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.613+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d080c5600 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.615+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d081551f0 con 0x7f3d0815bbb0 2024-03-19T02:54:53.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.615+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f3d0000ca00 con 0x7f3d0815bbb0 2024-03-19T02:54:53.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.616+0000 7f3cffe00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 0x7f3ce806e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:53.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.616+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3d0008a790 con 0x7f3d0815bbb0 2024-03-19T02:54:53.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.616+0000 7f3d07e00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 0x7f3ce806e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:53.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.617+0000 7f3d07e00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 0x7f3ce806e030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80012e0 tx=0x7f3cf8001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:53.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.620+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3d081551f0 con 0x7f3d0815bbb0 2024-03-19T02:54:53.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:53.822+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3d080c4c20 con 0x7f3d0815bbb0 2024-03-19T02:54:54.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.158+0000 7f3cffe00640 1 -- 172.21.15.92:0/1908366188 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f3d080c4c20 con 0x7f3d0815bbb0 2024-03-19T02:54:54.160 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_metadata' created 2024-03-19T02:54:54.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 msgr2=0x7f3ce806e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 0x7f3ce806e030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80012e0 tx=0x7f3cf8001630 comp rx=0 tx=0).stop 2024-03-19T02:54:54.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 msgr2=0x7f3d0814d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e960 tx=0x7f3d0000ee20 comp rx=0 tx=0).stop 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 shutdown_connections 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3ce806bb70 0x7f3ce806e030 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0815bbb0 0x7f3d0814d220 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080c81a0 0x7f3d0814cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 --2- 172.21.15.92:0/1908366188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080c7dc0 0x7f3d08151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.163+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 >> 172.21.15.92:0/1908366188 conn(0x7f3d080cd2a0 msgr2=0x7f3d081544f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.164+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 shutdown_connections 2024-03-19T02:54:54.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.164+0000 7f3d0ea00640 1 -- 172.21.15.92:0/1908366188 wait complete. 2024-03-19T02:54:54.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T02:54:54.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.386+0000 7f59e0800640 1 Processor -- start 2024-03-19T02:54:54.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.386+0000 7f59e0800640 1 -- start start 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59e0800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 0x7f59d815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59e0800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59e0800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d815f050 con 0x7f59d80cae10 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59e0800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f59d8067c50 con 0x7f59d8154ec0 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59e0800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59d81552a0 con 0x7f59d80cbe40 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59de800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59de800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46604/0 (socket says 172.21.15.92:46604) 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59de800640 1 -- 172.21.15.92:0/4151976879 learned_addr learned my addr 172.21.15.92:0/4151976879 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:54.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.387+0000 7f59dde00640 1 --2- 172.21.15.92:0/4151976879 >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 0x7f59d815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.388+0000 7f59de800640 1 -- 172.21.15.92:0/4151976879 >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 msgr2=0x7f59d815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.388+0000 7f59de800640 1 --2- 172.21.15.92:0/4151976879 >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 0x7f59d815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.388+0000 7f59de800640 1 -- 172.21.15.92:0/4151976879 >> v1:172.21.15.92:6789/0 conn(0x7f59d80cae10 legacy=0x7f59d80ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.388+0000 7f59de800640 1 -- 172.21.15.92:0/4151976879 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d80cc7d0 con 0x7f59d80cbe40 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.388+0000 7f59dde00640 1 --2- 172.21.15.92:0/4151976879 >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 0x7f59d815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:54:54.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.389+0000 7f59de800640 1 --2- 172.21.15.92:0/4151976879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009e50 tx=0x7f59d402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa5826fe59723cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.389+0000 7f59dca00640 1 -- 172.21.15.92:0/4151976879 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d402c9e0 con 0x7f59d80cbe40 2024-03-19T02:54:54.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59dca00640 1 -- 172.21.15.92:0/4151976879 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d402cb80 con 0x7f59d80cbe40 2024-03-19T02:54:54.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59dca00640 1 -- 172.21.15.92:0/4151976879 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40343f0 con 0x7f59d80cbe40 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 -- 172.21.15.92:0/4151976879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 msgr2=0x7f59d80cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 --2- 172.21.15.92:0/4151976879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009e50 tx=0x7f59d402be40 comp rx=0 tx=0).stop 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 -- 172.21.15.92:0/4151976879 shutdown_connections 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 --2- 172.21.15.92:0/4151976879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d80cbe40 0x7f59d80cc290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 --2- 172.21.15.92:0/4151976879 >> v2:172.21.15.92:3300/0 conn(0x7f59d8154ec0 0x7f59d815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 -- 172.21.15.92:0/4151976879 >> 172.21.15.92:0/4151976879 conn(0x7f59d806da20 msgr2=0x7f59d806de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 -- 172.21.15.92:0/4151976879 shutdown_connections 2024-03-19T02:54:54.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.390+0000 7f59e0800640 1 -- 172.21.15.92:0/4151976879 wait complete. 2024-03-19T02:54:54.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.392+0000 7f59e0800640 1 Processor -- start 2024-03-19T02:54:54.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.392+0000 7f59e0800640 1 -- start start 2024-03-19T02:54:54.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 0x7f59d815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 0x7f59d815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 0x7f59d81ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f59d815b1c0 con 0x7f59d80cbe40 2024-03-19T02:54:54.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f59d815aee0 con 0x7f59d80cae10 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59e0800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59d815b4a0 con 0x7f59d8154ec0 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 0x7f59d815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 0x7f59d815dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58074/0 (socket says 172.21.15.92:58074) 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59dd400640 1 -- 172.21.15.92:0/1322412789 learned_addr learned my addr 172.21.15.92:0/1322412789 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59de800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 0x7f59d81ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.393+0000 7f59dde00640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 0x7f59d815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 -- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 msgr2=0x7f59d815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 0x7f59d815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 -- 172.21.15.92:0/1322412789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 msgr2=0x7f59d815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 0x7f59d815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 -- 172.21.15.92:0/1322412789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d815be70 con 0x7f59d8154ec0 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59dde00640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 0x7f59d815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:54.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.394+0000 7f59de800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 0x7f59d81ee590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d4032000 tx=0x7f59d4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.395+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4043020 con 0x7f59d8154ec0 2024-03-19T02:54:54.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.395+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d403e450 con 0x7f59d8154ec0 2024-03-19T02:54:54.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.395+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d815e260 con 0x7f59d8154ec0 2024-03-19T02:54:54.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.395+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4009510 con 0x7f59d8154ec0 2024-03-19T02:54:54.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.395+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d815c3f0 con 0x7f59d8154ec0 2024-03-19T02:54:54.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.396+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d815c700 con 0x7f59d8154ec0 2024-03-19T02:54:54.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.397+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f59d4046020 con 0x7f59d8154ec0 2024-03-19T02:54:54.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.397+0000 7f59d2000640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.398+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f59d40ad0b0 con 0x7f59d8154ec0 2024-03-19T02:54:54.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.398+0000 7f59dde00640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.398+0000 7f59dde00640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00ae90 tx=0x7f59cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.402+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59d404d030 con 0x7f59d8154ec0 2024-03-19T02:54:54.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:54.608+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f59d80684c0 con 0x7f59d8154ec0 2024-03-19T02:54:55.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.168+0000 7f59d2000640 1 -- 172.21.15.92:0/1322412789 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f59d407b600 con 0x7f59d8154ec0 2024-03-19T02:54:55.171 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_data' created 2024-03-19T02:54:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.177+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 msgr2=0x7f59bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.177+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00ae90 tx=0x7f59cc008040 comp rx=0 tx=0).stop 2024-03-19T02:54:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.177+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 msgr2=0x7f59d81ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.177+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 0x7f59d81ee590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d4032000 tx=0x7f59d4004a40 comp rx=0 tx=0).stop 2024-03-19T02:54:55.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 shutdown_connections 2024-03-19T02:54:55.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00ae90 tx=0x7f59cc008040 comp rx=0 tx=0).stop 2024-03-19T02:54:55.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59d8154ec0 0x7f59d81ee590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59d80cbe40 0x7f59d815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 --2- 172.21.15.92:0/1322412789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59d80cae10 0x7f59d815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.178+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 >> 172.21.15.92:0/1322412789 conn(0x7f59d806da20 msgr2=0x7f59d80c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.181+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 shutdown_connections 2024-03-19T02:54:55.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.186+0000 7f59e0800640 1 -- 172.21.15.92:0/1322412789 wait complete. 2024-03-19T02:54:55.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T02:54:55.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.397+0000 7fa71be00640 1 Processor -- start 2024-03-19T02:54:55.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.397+0000 7fa71be00640 1 -- start start 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.398+0000 7fa71be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa71c0c6460 0x7fa71c1630f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.398+0000 7fa71be00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71c0c3830 con 0x7fa71c1519c0 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.398+0000 7fa71be00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71c0c5c90 con 0x7fa71c0c80f0 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.398+0000 7fa71be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa71c0c3db0 con 0x7fa71c0c6460 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71aa00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa71c1519c0 0x7fa71c071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48528/0 (socket says 172.21.15.92:48528) 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa71c0c6460 0x7fa71c1630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71aa00640 1 -- 172.21.15.92:0/1942581392 learned_addr learned my addr 172.21.15.92:0/1942581392 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:55.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71b400640 1 -- 172.21.15.92:0/1942581392 >> v1:172.21.15.159:6789/0 conn(0x7fa71c0c80f0 legacy=0x7fa71c06cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71b400640 1 -- 172.21.15.92:0/1942581392 >> v1:172.21.15.92:6789/0 conn(0x7fa71c1519c0 legacy=0x7fa71c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa719600640 1 -- 172.21.15.92:0/1942581392 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230432311 0 0) 0x7fa71c0c5c90 con 0x7fa71c0c80f0 2024-03-19T02:54:55.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.399+0000 7fa71b400640 1 -- 172.21.15.92:0/1942581392 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c0c3ad0 con 0x7fa71c0c6460 2024-03-19T02:54:55.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.400+0000 7fa71b400640 1 --2- 172.21.15.92:0/1942581392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1630f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa7140097a0 tx=0x7fa71402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1416aacf6bd3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa719600640 1 -- 172.21.15.92:0/1942581392 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa714037070 con 0x7fa71c0c6460 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa719600640 1 -- 172.21.15.92:0/1942581392 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa714030710 con 0x7fa71c0c6460 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa719600640 1 -- 172.21.15.92:0/1942581392 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa714030ae0 con 0x7fa71c0c6460 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 -- 172.21.15.92:0/1942581392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 msgr2=0x7fa71c1630f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 --2- 172.21.15.92:0/1942581392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1630f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa7140097a0 tx=0x7fa71402b790 comp rx=0 tx=0).stop 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 -- 172.21.15.92:0/1942581392 shutdown_connections 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 --2- 172.21.15.92:0/1942581392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1630f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 -- 172.21.15.92:0/1942581392 >> 172.21.15.92:0/1942581392 conn(0x7fa71c06bac0 msgr2=0x7fa71c0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 -- 172.21.15.92:0/1942581392 shutdown_connections 2024-03-19T02:54:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.401+0000 7fa71be00640 1 -- 172.21.15.92:0/1942581392 wait complete. 2024-03-19T02:54:55.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.403+0000 7fa71be00640 1 Processor -- start 2024-03-19T02:54:55.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.404+0000 7fa71be00640 1 -- start start 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 0x7fa71c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c072110 con 0x7fa71c0c6460 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c0713f0 con 0x7fa71c0c80f0 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71be00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa71c071040 con 0x7fa71c1519c0 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58100/0 (socket says 172.21.15.92:58100) 2024-03-19T02:54:55.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46632/0 (socket says 172.21.15.92:46632) 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71aa00640 1 -- 172.21.15.92:0/77702238 learned_addr learned my addr 172.21.15.92:0/77702238 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.405+0000 7fa71a000640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 0x7fa71c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 -- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 msgr2=0x7fa71c14cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 0x7fa71c14cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 -- 172.21.15.92:0/77702238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 msgr2=0x7fa71c1478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 -- 172.21.15.92:0/77702238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c146ad0 con 0x7fa71c1519c0 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71a000640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 0x7fa71c14cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71aa00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71b400640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa714032000 tx=0x7fa7140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa714042070 con 0x7fa71c1519c0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7140329c0 con 0x7fa71c1519c0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa714037040 con 0x7fa71c1519c0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71c14e940 con 0x7fa71c1519c0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.406+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71c14bcc0 con 0x7fa71c1519c0 2024-03-19T02:54:55.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.408+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71c04f580 con 0x7fa71c1519c0 2024-03-19T02:54:55.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.409+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa714045070 con 0x7fa71c1519c0 2024-03-19T02:54:55.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.409+0000 7fa713400640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 0x7fa6f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.409+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fa7140ac760 con 0x7fa71c1519c0 2024-03-19T02:54:55.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.410+0000 7fa71aa00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 0x7fa6f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.410+0000 7fa71aa00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 0x7fa6f806e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa70c002230 tx=0x7fa70c0018f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.414+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa71407ae10 con 0x7fa71c1519c0 2024-03-19T02:54:55.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:55.652+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa71c0c8a00 con 0x7fa71c1519c0 2024-03-19T02:54:56.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.177+0000 7fa713400640 1 -- 172.21.15.92:0/77702238 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fa71407eeb0 con 0x7fa71c1519c0 2024-03-19T02:54:56.184 INFO:teuthology.orchestra.run.smithi092.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T02:54:56.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.185+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 msgr2=0x7fa6f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.185+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 0x7fa6f806e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa70c002230 tx=0x7fa70c0018f0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.185+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 msgr2=0x7fa71c1ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.185+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa714032000 tx=0x7fa7140046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 shutdown_connections 2024-03-19T02:54:56.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa6f806bbd0 0x7fa6f806e090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa71c1519c0 0x7fa71c1ee560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa71c0c80f0 0x7fa71c14cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 --2- 172.21.15.92:0/77702238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa71c0c6460 0x7fa71c1478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.186+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 >> 172.21.15.92:0/77702238 conn(0x7fa71c06bac0 msgr2=0x7fa71c0cc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.187+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 shutdown_connections 2024-03-19T02:54:56.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.187+0000 7fa71be00640 1 -- 172.21.15.92:0/77702238 wait complete. 2024-03-19T02:54:56.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:56.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.398+0000 7f0245600640 1 Processor -- start 2024-03-19T02:54:56.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.398+0000 7f0245600640 1 -- start start 2024-03-19T02:54:56.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f0245600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f02401550b0 0x7f024015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f0245600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0240067ad0 0x7f02400cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f0245600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02400caf90 con 0x7f0240154cd0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f0245600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f024015e280 con 0x7f0240067ad0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f0245600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f024015e520 con 0x7f02401550b0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0240067ad0 0x7f02400cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0240154cd0 0x7f02400cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57844/0 (socket says 172.21.15.92:57844) 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023ea00640 1 -- 172.21.15.92:0/2118520309 learned_addr learned my addr 172.21.15.92:0/2118520309 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023e000640 1 --2- 172.21.15.92:0/2118520309 >> v2:172.21.15.159:3300/0 conn(0x7f02401550b0 0x7f024015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023f400640 1 -- 172.21.15.92:0/2118520309 >> v2:172.21.15.159:3300/0 conn(0x7f02401550b0 msgr2=0x7f024015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023f400640 1 --2- 172.21.15.92:0/2118520309 >> v2:172.21.15.159:3300/0 conn(0x7f02401550b0 0x7f024015ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.400+0000 7f023f400640 1 -- 172.21.15.92:0/2118520309 >> v1:172.21.15.92:6789/0 conn(0x7f0240154cd0 legacy=0x7f02400cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.401+0000 7f023f400640 1 -- 172.21.15.92:0/2118520309 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024015e800 con 0x7f0240067ad0 2024-03-19T02:54:56.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.401+0000 7f023d600640 1 -- 172.21.15.92:0/2118520309 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288161853 0 0) 0x7f0230001110 con 0x7f0240154cd0 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f023f400640 1 --2- 172.21.15.92:0/2118520309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240067ad0 0x7f02400cbfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f02400cdfa0 tx=0x7f023802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914fdb16de347a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f023d600640 1 -- 172.21.15.92:0/2118520309 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238039070 con 0x7f0240067ad0 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f023d600640 1 -- 172.21.15.92:0/2118520309 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f023802cc80 con 0x7f0240067ad0 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f023d600640 1 -- 172.21.15.92:0/2118520309 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238034660 con 0x7f0240067ad0 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 -- 172.21.15.92:0/2118520309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240067ad0 msgr2=0x7f02400cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 --2- 172.21.15.92:0/2118520309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240067ad0 0x7f02400cbfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f02400cdfa0 tx=0x7f023802bed0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 -- 172.21.15.92:0/2118520309 shutdown_connections 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 --2- 172.21.15.92:0/2118520309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240067ad0 0x7f02400cbfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 --2- 172.21.15.92:0/2118520309 >> v2:172.21.15.159:3300/0 conn(0x7f02401550b0 0x7f024015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.402+0000 7f0245600640 1 -- 172.21.15.92:0/2118520309 >> 172.21.15.92:0/2118520309 conn(0x7f0240072970 msgr2=0x7f0240066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.403+0000 7f0245600640 1 -- 172.21.15.92:0/2118520309 shutdown_connections 2024-03-19T02:54:56.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.403+0000 7f0245600640 1 -- 172.21.15.92:0/2118520309 wait complete. 2024-03-19T02:54:56.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.404+0000 7f0245600640 1 Processor -- start 2024-03-19T02:54:56.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.404+0000 7f0245600640 1 -- start start 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0240067ad0 0x7f02401f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02401550b0 0x7f024014c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02400cd5c0 con 0x7f0240154cd0 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f024015edc0 con 0x7f0240067ad0 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f0245600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f02400cd740 con 0x7f02401550b0 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f023e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f023e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47702/0 (socket says 172.21.15.92:47702) 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f023e000640 1 -- 172.21.15.92:0/2367843670 learned_addr learned my addr 172.21.15.92:0/2367843670 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.405+0000 7f023ea00640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0240067ad0 0x7f02401f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023f400640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02401550b0 0x7f024014c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 -- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02401550b0 msgr2=0x7f024014c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02401550b0 0x7f024014c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 -- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0240067ad0 msgr2=0x7f02401f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0240067ad0 0x7f02401f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 -- 172.21.15.92:0/2367843670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02401f3a50 con 0x7f0240154cd0 2024-03-19T02:54:56.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f023e000640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f022800e870 tx=0x7f022800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228017070 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022800c940 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022800cd30 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.406+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02400b44d0 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.407+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02401f3fd0 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.408+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02400684c0 con 0x7f0240154cd0 2024-03-19T02:54:56.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.408+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f022801a070 con 0x7f0240154cd0 2024-03-19T02:54:56.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.409+0000 7f0236a00640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 0x7f021c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.409+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f022808a9a0 con 0x7f0240154cd0 2024-03-19T02:54:56.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.410+0000 7f023ea00640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 0x7f021c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.410+0000 7f023ea00640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 0x7f021c06e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0230007fd0 tx=0x7f0230007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.414+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0228058d10 con 0x7f0240154cd0 2024-03-19T02:54:56.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.613+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02400ca8f0 con 0x7f0240154cd0 2024-03-19T02:54:56.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.616+0000 7f0236a00640 1 -- 172.21.15.92:0/2367843670 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16618 (secure 0 0 0) 0x7f022805cdb0 con 0x7f0240154cd0 2024-03-19T02:54:56.618 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:56.619 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:55.655495+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 msgr2=0x7f021c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 0x7f021c06e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0230007fd0 tx=0x7f0230007de0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 msgr2=0x7f024015c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f022800e870 tx=0x7f022800ed30 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 shutdown_connections 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f021c06bbd0 0x7f021c06e090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02401550b0 0x7f024014c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0240154cd0 0x7f024015c160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 --2- 172.21.15.92:0/2367843670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0240067ad0 0x7f02401f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.619+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 >> 172.21.15.92:0/2367843670 conn(0x7f0240072970 msgr2=0x7f024014d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.620+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 shutdown_connections 2024-03-19T02:54:56.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.620+0000 7f0245600640 1 -- 172.21.15.92:0/2367843670 wait complete. 2024-03-19T02:54:56.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:54:56.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.844+0000 7f9ab9400640 1 Processor -- start 2024-03-19T02:54:56.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.844+0000 7f9ab9400640 1 -- start start 2024-03-19T02:54:56.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab9400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9ab4152bf0 0x7f9ab406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab9400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 0x7f9ab40c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab9400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab40c9a80 con 0x7f9ab41593d0 2024-03-19T02:54:56.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab9400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab40c6c90 con 0x7f9ab4152bf0 2024-03-19T02:54:56.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab9400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab4159910 con 0x7f9ab40c6470 2024-03-19T02:54:56.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab2800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9ab4152bf0 0x7f9ab406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab3200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9ab41593d0 0x7f9ab40c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46800/0 (socket says 172.21.15.92:46800) 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9aabe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 0x7f9ab40c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.846+0000 7f9ab3200640 1 -- 172.21.15.92:0/1538114067 learned_addr learned my addr 172.21.15.92:0/1538114067 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9ab1e00640 1 -- 172.21.15.92:0/1538114067 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551393212 0 0) 0x7f9ab40c9a80 con 0x7f9ab41593d0 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9ab2800640 1 -- 172.21.15.92:0/1538114067 >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 msgr2=0x7f9ab40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9ab2800640 1 --2- 172.21.15.92:0/1538114067 >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 0x7f9ab40c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9ab2800640 1 -- 172.21.15.92:0/1538114067 >> v1:172.21.15.159:6789/0 conn(0x7f9ab41593d0 legacy=0x7f9ab40c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9ab2800640 1 -- 172.21.15.92:0/1538114067 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab40c8650 con 0x7f9ab4152bf0 2024-03-19T02:54:56.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.847+0000 7f9aabe00640 1 --2- 172.21.15.92:0/1538114067 >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 0x7f9ab40c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab2800640 1 --2- 172.21.15.92:0/1538114067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab406eef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4075340 tx=0x7f9aa402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78c84de76a013e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab1e00640 1 -- 172.21.15.92:0/1538114067 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40315a0 con 0x7f9ab4152bf0 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab1e00640 1 -- 172.21.15.92:0/1538114067 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa4037070 con 0x7f9ab4152bf0 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab1e00640 1 -- 172.21.15.92:0/1538114067 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4031960 con 0x7f9ab4152bf0 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab9400640 1 -- 172.21.15.92:0/1538114067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 msgr2=0x7f9ab406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.848+0000 7f9ab9400640 1 --2- 172.21.15.92:0/1538114067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab406eef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4075340 tx=0x7f9aa402b840 comp rx=0 tx=0).stop 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 -- 172.21.15.92:0/1538114067 shutdown_connections 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 --2- 172.21.15.92:0/1538114067 >> v2:172.21.15.159:3300/0 conn(0x7f9ab40c6470 0x7f9ab40c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 --2- 172.21.15.92:0/1538114067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab406eef0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 -- 172.21.15.92:0/1538114067 >> 172.21.15.92:0/1538114067 conn(0x7f9ab40c8d90 msgr2=0x7f9ab40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 -- 172.21.15.92:0/1538114067 shutdown_connections 2024-03-19T02:54:56.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.849+0000 7f9ab9400640 1 -- 172.21.15.92:0/1538114067 wait complete. 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.850+0000 7f9ab9400640 1 Processor -- start 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.851+0000 7f9ab9400640 1 -- start start 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.851+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab40c6470 0x7f9ab4148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ab41593d0 0x7f9ab41f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab9400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab406ea60 con 0x7f9ab4152bf0 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab9400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab406e4a0 con 0x7f9ab40c6470 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab9400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab406e6f0 con 0x7f9ab41593d0 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47724/0 (socket says 172.21.15.92:47724) 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 -- 172.21.15.92:0/3033626902 learned_addr learned my addr 172.21.15.92:0/3033626902 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab2800640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab40c6470 0x7f9ab4148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9ab3200640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ab41593d0 0x7f9ab41f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 -- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ab41593d0 msgr2=0x7f9ab41f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ab41593d0 0x7f9ab41f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 -- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab40c6470 msgr2=0x7f9ab4148090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab40c6470 0x7f9ab4148090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.852+0000 7f9aabe00640 1 -- 172.21.15.92:0/3033626902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab41ef820 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9aabe00640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00ebe0 tx=0x7f9a9c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c017070 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00cc30 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab41eec10 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c013470 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.853+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab41ef1d0 con 0x7f9ab4152bf0 2024-03-19T02:54:56.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.854+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab4152370 con 0x7f9ab4152bf0 2024-03-19T02:54:56.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.855+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9a9c01a070 con 0x7f9ab4152bf0 2024-03-19T02:54:56.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.856+0000 7f9aaaa00640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 0x7f9a9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.856+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f9a9c08a940 con 0x7f9ab4152bf0 2024-03-19T02:54:56.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.856+0000 7f9ab2800640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 0x7f9a9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.857+0000 7f9ab2800640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 0x7f9a9006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4036000 tx=0x7f9aa4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:56.861+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9ab4152370 con 0x7f9ab4152bf0 2024-03-19T02:54:57.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.043+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ab4068de0 con 0x7f9ab4152bf0 2024-03-19T02:54:57.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.045+0000 7f9aaaa00640 1 -- 172.21.15.92:0/3033626902 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16618 (secure 0 0 0) 0x7f9ab4068de0 con 0x7f9ab4152bf0 2024-03-19T02:54:57.046 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:54:57.047 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:55.655495+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:54:57.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 msgr2=0x7f9a9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 0x7f9a9006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4036000 tx=0x7f9aa4008030 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 msgr2=0x7f9ab41f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00ebe0 tx=0x7f9a9c00c520 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 shutdown_connections 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9a9006bbd0 0x7f9a9006e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ab41593d0 0x7f9ab41f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ab4152bf0 0x7f9ab41f2170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 --2- 172.21.15.92:0/3033626902 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ab40c6470 0x7f9ab4148090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 >> 172.21.15.92:0/3033626902 conn(0x7f9ab40c8d90 msgr2=0x7f9ab40c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 shutdown_connections 2024-03-19T02:54:57.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.047+0000 7f9ab9400640 1 -- 172.21.15.92:0/3033626902 wait complete. 2024-03-19T02:54:57.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T02:54:57.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.277+0000 7f39cf000640 1 Processor -- start 2024-03-19T02:54:57.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.277+0000 7f39cf000640 1 -- start start 2024-03-19T02:54:57.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.278+0000 7f39cf000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c80c6470 0x7f39c80c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.278+0000 7f39cf000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f39c81593d0 0x7f39c80c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cf000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c80c9a80 con 0x7f39c8152bf0 2024-03-19T02:54:57.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cf000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f39c80c6c90 con 0x7f39c81593d0 2024-03-19T02:54:57.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cf000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f39c8159910 con 0x7f39c80c6470 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39c7e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f39c8152bf0 0x7f39c806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46802/0 (socket says 172.21.15.92:46802) 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39c7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c80c6470 0x7f39c80c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39c7e00640 1 -- 172.21.15.92:0/6299756 learned_addr learned my addr 172.21.15.92:0/6299756 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cd000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f39c81593d0 0x7f39c80c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cd000640 1 -- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c80c6470 msgr2=0x7f39c80c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:57.281 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.279+0000 7f39cd000640 1 --2- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c80c6470 0x7f39c80c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.282 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.280+0000 7f39cd000640 1 -- 172.21.15.92:0/6299756 >> v1:172.21.15.159:6789/0 conn(0x7f39c8152bf0 legacy=0x7f39c806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.282 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.280+0000 7f39cd000640 1 -- 172.21.15.92:0/6299756 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c80c8650 con 0x7f39c81593d0 2024-03-19T02:54:57.282 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.280+0000 7f39c6a00640 1 -- 172.21.15.92:0/6299756 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712088279 0 0) 0x7f39c80c9a80 con 0x7f39c8152bf0 2024-03-19T02:54:57.283 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cd000640 1 --2- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c81593d0 0x7f39c80c44c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097a0 tx=0x7f39c002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b2ca74af4a8638 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39c6a00640 1 -- 172.21.15.92:0/6299756 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00315a0 con 0x7f39c81593d0 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39c6a00640 1 -- 172.21.15.92:0/6299756 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c0037070 con 0x7f39c81593d0 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39c6a00640 1 -- 172.21.15.92:0/6299756 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0031960 con 0x7f39c81593d0 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 -- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c81593d0 msgr2=0x7f39c80c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 --2- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c81593d0 0x7f39c80c44c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097a0 tx=0x7f39c002b840 comp rx=0 tx=0).stop 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 -- 172.21.15.92:0/6299756 shutdown_connections 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 --2- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c80c6470 0x7f39c80c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 --2- 172.21.15.92:0/6299756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c81593d0 0x7f39c80c44c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 -- 172.21.15.92:0/6299756 >> 172.21.15.92:0/6299756 conn(0x7f39c80c8d90 msgr2=0x7f39c80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:57.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 -- 172.21.15.92:0/6299756 shutdown_connections 2024-03-19T02:54:57.285 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.281+0000 7f39cf000640 1 -- 172.21.15.92:0/6299756 wait complete. 2024-03-19T02:54:57.287 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.283+0000 7f39cf000640 1 Processor -- start 2024-03-19T02:54:57.287 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.283+0000 7f39cf000640 1 -- start start 2024-03-19T02:54:57.287 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 0x7f39c80cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.287 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c8152bf0 0x7f39c8154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f39c806ea60 con 0x7f39c81593d0 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f39c806e4a0 con 0x7f39c80c6470 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cf000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f39c806e6f0 con 0x7f39c8152bf0 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47736/0 (socket says 172.21.15.92:47736) 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39c7e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 0x7f39c80cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39cd000640 1 -- 172.21.15.92:0/3497173172 learned_addr learned my addr 172.21.15.92:0/3497173172 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.284+0000 7f39c7400640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c8152bf0 0x7f39c8154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 -- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c8152bf0 msgr2=0x7f39c8154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c8152bf0 0x7f39c8154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 -- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 msgr2=0x7f39c80cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 0x7f39c80cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 -- 172.21.15.92:0/3497173172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81f2630 con 0x7f39c81593d0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39c7e00640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 0x7f39c80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39cd000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39c00098c0 tx=0x7f39c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.285+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0032630 con 0x7f39c81593d0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.286+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c00327d0 con 0x7f39c81593d0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.286+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0032bc0 con 0x7f39c81593d0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.286+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c814eda0 con 0x7f39c81593d0 2024-03-19T02:54:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.286+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c814eac0 con 0x7f39c81593d0 2024-03-19T02:54:57.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.287+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c80684c0 con 0x7f39c81593d0 2024-03-19T02:54:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.288+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f39c004c020 con 0x7f39c81593d0 2024-03-19T02:54:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.288+0000 7f39bfe00640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 0x7f39a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.288+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f39c002f020 con 0x7f39c81593d0 2024-03-19T02:54:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.289+0000 7f39c7e00640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 0x7f39a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:57.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.289+0000 7f39c7e00640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 0x7f39a406e0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39b0000f60 tx=0x7f39b0007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:57.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.293+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f39c0040070 con 0x7f39c81593d0 2024-03-19T02:54:57.516 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:57.515+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f39c806f950 con 0x7f39c81593d0 2024-03-19T02:54:58.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.200+0000 7f39bfe00640 1 -- 172.21.15.92:0/3497173172 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f39c007aa00 con 0x7f39c81593d0 2024-03-19T02:54:58.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 msgr2=0x7f39a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 0x7f39a406e0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39b0000f60 tx=0x7f39b0007d50 comp rx=0 tx=0).stop 2024-03-19T02:54:58.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 msgr2=0x7f39c8142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39c00098c0 tx=0x7f39c00046c0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 shutdown_connections 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39a406bc00 0x7f39a406e0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39c81593d0 0x7f39c8142bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c8152bf0 0x7f39c8154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 --2- 172.21.15.92:0/3497173172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c80c6470 0x7f39c80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.203+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 >> 172.21.15.92:0/3497173172 conn(0x7f39c80c8d90 msgr2=0x7f39c806eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.204+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 shutdown_connections 2024-03-19T02:54:58.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.204+0000 7f39cf000640 1 -- 172.21.15.92:0/3497173172 wait complete. 2024-03-19T02:54:58.226 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-03-19T02:54:58.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-19T02:54:58.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:54:58.360 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:54:58.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.431+0000 7fdf99a00640 1 Processor -- start 2024-03-19T02:54:58.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.431+0000 7fdf99a00640 1 -- start start 2024-03-19T02:54:58.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.432+0000 7fdf99a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf940c9da0 0x7fdf940ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:58.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.432+0000 7fdf99a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9406bf20 con 0x7fdf94154d10 2024-03-19T02:54:58.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.432+0000 7fdf99a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9406c1c0 con 0x7fdf9406b240 2024-03-19T02:54:58.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.432+0000 7fdf99a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdf9406f360 con 0x7fdf940c9da0 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf99000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf940c9da0 0x7fdf940ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf93e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdf94154d10 0x7fdf940c7d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57852/0 (socket says 172.21.15.92:57852) 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf93e00640 1 -- 172.21.15.92:0/2356995648 learned_addr learned my addr 172.21.15.92:0/2356995648 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf99000640 1 -- 172.21.15.92:0/2356995648 >> v1:172.21.15.159:6789/0 conn(0x7fdf9406b240 legacy=0x7fdf9415b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf99000640 1 -- 172.21.15.92:0/2356995648 >> v1:172.21.15.92:6789/0 conn(0x7fdf94154d10 legacy=0x7fdf940c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf99000640 1 -- 172.21.15.92:0/2356995648 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9406bc50 con 0x7fdf940c9da0 2024-03-19T02:54:58.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.433+0000 7fdf92a00640 1 -- 172.21.15.92:0/2356995648 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472979841 0 0) 0x7fdf9406c1c0 con 0x7fdf9406b240 2024-03-19T02:54:58.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.434+0000 7fdf99000640 1 --2- 172.21.15.92:0/2356995648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf940c9da0 0x7fdf940ca1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0097a0 tx=0x7fdf8c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5f3509e9aa26c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:58.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.434+0000 7fdf92a00640 1 -- 172.21.15.92:0/2356995648 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c02f810 con 0x7fdf940c9da0 2024-03-19T02:54:58.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.434+0000 7fdf92a00640 1 -- 172.21.15.92:0/2356995648 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf8c02f9b0 con 0x7fdf940c9da0 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.434+0000 7fdf92a00640 1 -- 172.21.15.92:0/2356995648 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c02fd80 con 0x7fdf940c9da0 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 -- 172.21.15.92:0/2356995648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf940c9da0 msgr2=0x7fdf940ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 --2- 172.21.15.92:0/2356995648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf940c9da0 0x7fdf940ca1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0097a0 tx=0x7fdf8c02b9f0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 -- 172.21.15.92:0/2356995648 shutdown_connections 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 --2- 172.21.15.92:0/2356995648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf940c9da0 0x7fdf940ca1a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 -- 172.21.15.92:0/2356995648 >> 172.21.15.92:0/2356995648 conn(0x7fdf94072030 msgr2=0x7fdf94072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 -- 172.21.15.92:0/2356995648 shutdown_connections 2024-03-19T02:54:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.435+0000 7fdf99a00640 1 -- 172.21.15.92:0/2356995648 wait complete. 2024-03-19T02:54:58.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.437+0000 7fdf99a00640 1 Processor -- start 2024-03-19T02:54:58.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.437+0000 7fdf99a00640 1 -- start start 2024-03-19T02:54:58.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf9406b240 0x7fdf9414a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:58.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf940c9da0 0x7fdf9414fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9406c760 con 0x7fdf94154d10 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9406ca40 con 0x7fdf9406b240 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9415ab60 con 0x7fdf940c9da0 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf93e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf9406b240 0x7fdf9414a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47768/0 (socket says 172.21.15.92:47768) 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 -- 172.21.15.92:0/3542904057 learned_addr learned my addr 172.21.15.92:0/3542904057 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf93400640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf940c9da0 0x7fdf9414fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 -- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf940c9da0 msgr2=0x7fdf9414fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf940c9da0 0x7fdf9414fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 -- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf9406b240 msgr2=0x7fdf9414a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf9406b240 0x7fdf9414a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.438+0000 7fdf99000640 1 -- 172.21.15.92:0/3542904057 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf941f4b70 con 0x7fdf94154d10 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf99000640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c02c000 tx=0x7fdf8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c046070 con 0x7fdf94154d10 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf8c037bc0 con 0x7fdf94154d10 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf940c51f0 con 0x7fdf94154d10 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c02fe60 con 0x7fdf94154d10 2024-03-19T02:54:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.439+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf940c57b0 con 0x7fdf94154d10 2024-03-19T02:54:58.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.440+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf941540d0 con 0x7fdf94154d10 2024-03-19T02:54:58.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.441+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdf8c049070 con 0x7fdf94154d10 2024-03-19T02:54:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.442+0000 7fdf8be00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.442+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdf8c0ac560 con 0x7fdf94154d10 2024-03-19T02:54:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.443+0000 7fdf93e00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.443+0000 7fdf93e00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 0x7fdf7406e090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf84007fd0 tx=0x7fdf84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:58.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.446+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdf941540d0 con 0x7fdf94154d10 2024-03-19T02:54:58.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:58.685+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdf9404f580 con 0x7fdf94154d10 2024-03-19T02:54:59.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.210+0000 7fdf8be00640 1 -- 172.21.15.92:0/3542904057 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7fdf8c033030 con 0x7fdf94154d10 2024-03-19T02:54:59.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.213+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 msgr2=0x7fdf7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.213+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 0x7fdf7406e090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf84007fd0 tx=0x7fdf84009040 comp rx=0 tx=0).stop 2024-03-19T02:54:59.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.213+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 msgr2=0x7fdf94144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.213+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c02c000 tx=0x7fdf8c004730 comp rx=0 tx=0).stop 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 shutdown_connections 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf94154d10 0x7fdf94144d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf940c9da0 0x7fdf9414fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 --2- 172.21.15.92:0/3542904057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf9406b240 0x7fdf9414a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 >> 172.21.15.92:0/3542904057 conn(0x7fdf94072030 msgr2=0x7fdf9415af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:59.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 shutdown_connections 2024-03-19T02:54:59.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.214+0000 7fdf99a00640 1 -- 172.21.15.92:0/3542904057 wait complete. 2024-03-19T02:54:59.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T02:54:59.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.442+0000 7fe698e00640 1 Processor -- start 2024-03-19T02:54:59.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.442+0000 7fe698e00640 1 -- start start 2024-03-19T02:54:59.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe698e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe694158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe698e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 0x7fe6940c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe698e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940c3b80 con 0x7fe694151ce0 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe698e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fe694158860 con 0x7fe6940c7da0 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe698e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe694072e70 con 0x7fe694151900 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe689400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe694151ce0 0x7fe6940723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57864/0 (socket says 172.21.15.92:57864) 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe692800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 0x7fe6940c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe691e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe694158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.444+0000 7fe689400640 1 -- 172.21.15.92:0/1742608695 learned_addr learned my addr 172.21.15.92:0/1742608695 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe691e00640 1 -- 172.21.15.92:0/1742608695 >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 msgr2=0x7fe6940c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe691e00640 1 --2- 172.21.15.92:0/1742608695 >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 0x7fe6940c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe691e00640 1 -- 172.21.15.92:0/1742608695 >> v1:172.21.15.92:6789/0 conn(0x7fe694151ce0 legacy=0x7fe6940723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe691e00640 1 -- 172.21.15.92:0/1742608695 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe694073150 con 0x7fe694151900 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe692800640 1 --2- 172.21.15.92:0/1742608695 >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 0x7fe6940c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:59.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.445+0000 7fe691400640 1 -- 172.21.15.92:0/1742608695 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035198348 0 0) 0x7fe6940c3b80 con 0x7fe694151ce0 2024-03-19T02:54:59.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.446+0000 7fe691e00640 1 --2- 172.21.15.92:0/1742608695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe694158320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c425cdfa0b6ddfc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:59.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.446+0000 7fe691400640 1 -- 172.21.15.92:0/1742608695 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684034070 con 0x7fe694151900 2024-03-19T02:54:59.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.446+0000 7fe691400640 1 -- 172.21.15.92:0/1742608695 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe684030800 con 0x7fe694151900 2024-03-19T02:54:59.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.446+0000 7fe691400640 1 -- 172.21.15.92:0/1742608695 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684030bd0 con 0x7fe694151900 2024-03-19T02:54:59.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 -- 172.21.15.92:0/1742608695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 msgr2=0x7fe694158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 --2- 172.21.15.92:0/1742608695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe694158320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b940 comp rx=0 tx=0).stop 2024-03-19T02:54:59.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 -- 172.21.15.92:0/1742608695 shutdown_connections 2024-03-19T02:54:59.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 --2- 172.21.15.92:0/1742608695 >> v2:172.21.15.159:3300/0 conn(0x7fe6940c7da0 0x7fe6940c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 --2- 172.21.15.92:0/1742608695 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe694158320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 -- 172.21.15.92:0/1742608695 >> 172.21.15.92:0/1742608695 conn(0x7fe6940c8d90 msgr2=0x7fe6940c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:59.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 -- 172.21.15.92:0/1742608695 shutdown_connections 2024-03-19T02:54:59.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.447+0000 7fe698e00640 1 -- 172.21.15.92:0/1742608695 wait complete. 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.448+0000 7fe698e00640 1 Processor -- start 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 -- start start 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6940c7da0 0x7fe694164db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe6941645d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940c9af0 con 0x7fe694151ce0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe694071eb0 con 0x7fe6940c7da0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.449+0000 7fe698e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe6940c9c70 con 0x7fe694151900 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe689400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe6941645d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe691e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6940c7da0 0x7fe694164db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47774/0 (socket says 172.21.15.92:47774) 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 -- 172.21.15.92:0/4057215443 learned_addr learned my addr 172.21.15.92:0/4057215443 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 -- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 msgr2=0x7fe6941645d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe6941645d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 -- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6940c7da0 msgr2=0x7fe694164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6940c7da0 0x7fe694164db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 -- 172.21.15.92:0/4057215443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe694145820 con 0x7fe694151ce0 2024-03-19T02:54:59.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe689400640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe6941645d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.450+0000 7fe692800640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00efc0 tx=0x7fe68c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.451+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c018030 con 0x7fe694151ce0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.451+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c00cdf0 con 0x7fe694151ce0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.451+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c003bb0 con 0x7fe694151ce0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.451+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe694145b00 con 0x7fe694151ce0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.451+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6941f3e90 con 0x7fe694151ce0 2024-03-19T02:54:59.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.452+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe694146020 con 0x7fe694151ce0 2024-03-19T02:54:59.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.453+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe68c01b070 con 0x7fe694151ce0 2024-03-19T02:54:59.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.453+0000 7fe68aa00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 0x7fe67006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:59.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.454+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe68c08b340 con 0x7fe694151ce0 2024-03-19T02:54:59.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.454+0000 7fe691e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 0x7fe67006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:59.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.455+0000 7fe691e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 0x7fe67006e090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe684008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:59.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.458+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe68c010040 con 0x7fe694151ce0 2024-03-19T02:54:59.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:54:59.694+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fe6941f3c80 con 0x7fe694151ce0 2024-03-19T02:55:00.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.218+0000 7fe68aa00640 1 -- 172.21.15.92:0/4057215443 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) ==== 115+0+0 (secure 0 0 0) 0x7fe68c059740 con 0x7fe694151ce0 2024-03-19T02:55:00.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 msgr2=0x7fe67006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 0x7fe67006e090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe684008030 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 msgr2=0x7fe6941f4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00efc0 tx=0x7fe68c00c490 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 shutdown_connections 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe67006bbd0 0x7fe67006e090 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe694151ce0 0x7fe6941f4690 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe694151900 0x7fe6941645d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 --2- 172.21.15.92:0/4057215443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6940c7da0 0x7fe694164db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 >> 172.21.15.92:0/4057215443 conn(0x7fe6940c8d90 msgr2=0x7fe6940cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.222+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 shutdown_connections 2024-03-19T02:55:00.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.223+0000 7fe698e00640 1 -- 172.21.15.92:0/4057215443 wait complete. 2024-03-19T02:55:00.244 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-19T02:55:00.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:00.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.452+0000 7f88ada00640 1 Processor -- start 2024-03-19T02:55:00.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.452+0000 7f88ada00640 1 -- start start 2024-03-19T02:55:00.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.453+0000 7f88ada00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.453+0000 7f88ada00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88a80c6470 0x7f88a80c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.453+0000 7f88ada00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a80c9a80 con 0x7f88a81593d0 2024-03-19T02:55:00.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.453+0000 7f88ada00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f88a80c6c90 con 0x7f88a80c6470 2024-03-19T02:55:00.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.453+0000 7f88ada00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f88a8159910 con 0x7f88a8152bf0 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88a80c6470 0x7f88a80c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a7400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f88a81593d0 0x7f88a80c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46822/0 (socket says 172.21.15.92:46822) 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a7400640 1 -- 172.21.15.92:0/3795004111 learned_addr learned my addr 172.21.15.92:0/3795004111 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6a00640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6000640 1 -- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 msgr2=0x7f88a806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6000640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6000640 1 -- 172.21.15.92:0/3795004111 >> v1:172.21.15.159:6789/0 conn(0x7f88a81593d0 legacy=0x7f88a80c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.454+0000 7f88a6000640 1 -- 172.21.15.92:0/3795004111 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a80c8650 con 0x7f88a80c6470 2024-03-19T02:55:00.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.455+0000 7f88a6000640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a80c6470 0x7f88a80c3f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88940097a0 tx=0x7f889402b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f84f6a14ea4139e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.455+0000 7f88a5600640 1 -- 172.21.15.92:0/3795004111 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894037070 con 0x7f88a80c6470 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.455+0000 7f88a5600640 1 -- 172.21.15.92:0/3795004111 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894030710 con 0x7f88a80c6470 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.455+0000 7f88a5600640 1 -- 172.21.15.92:0/3795004111 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889402be60 con 0x7f88a80c6470 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 -- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a80c6470 msgr2=0x7f88a80c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a80c6470 0x7f88a80c3f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88940097a0 tx=0x7f889402b660 comp rx=0 tx=0).stop 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 -- 172.21.15.92:0/3795004111 shutdown_connections 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a80c6470 0x7f88a80c3f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 --2- 172.21.15.92:0/3795004111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 -- 172.21.15.92:0/3795004111 >> 172.21.15.92:0/3795004111 conn(0x7f88a80c8d90 msgr2=0x7f88a80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 -- 172.21.15.92:0/3795004111 shutdown_connections 2024-03-19T02:55:00.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.456+0000 7f88ada00640 1 -- 172.21.15.92:0/3795004111 wait complete. 2024-03-19T02:55:00.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.458+0000 7f88ada00640 1 Processor -- start 2024-03-19T02:55:00.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.458+0000 7f88ada00640 1 -- start start 2024-03-19T02:55:00.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a81593d0 0x7f88a81eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88a806ea60 con 0x7f88a80c6470 2024-03-19T02:55:00.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f88a806e4a0 con 0x7f88a81593d0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88ada00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f88a806e6f0 con 0x7f88a8152bf0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47784/0 (socket says 172.21.15.92:47784) 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 -- 172.21.15.92:0/3628282742 learned_addr learned my addr 172.21.15.92:0/3628282742 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a7400640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a81593d0 0x7f88a81eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6000640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 -- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 msgr2=0x7f88a814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a814e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 -- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a81593d0 msgr2=0x7f88a81eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a81593d0 0x7f88a81eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.459+0000 7f88a6a00640 1 -- 172.21.15.92:0/3628282742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a81f2350 con 0x7f88a80c6470 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f88a6a00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f889800e9b0 tx=0x7f889800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898017070 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889800c9a0 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889800cd90 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a81f2630 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.460+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a81f2bf0 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.461+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a80684c0 con 0x7f88a80c6470 2024-03-19T02:55:00.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.462+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f889801a070 con 0x7f88a80c6470 2024-03-19T02:55:00.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.463+0000 7f889ea00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.463+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f889808aad0 con 0x7f88a80c6470 2024-03-19T02:55:00.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.463+0000 7f88a6000640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.464+0000 7f88a6000640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 0x7f888806e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8894034000 tx=0x7f8894008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.467+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8898059530 con 0x7f88a80c6470 2024-03-19T02:55:00.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.685+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a8150970 con 0x7f88a80c6470 2024-03-19T02:55:00.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.687+0000 7f889ea00640 1 -- 172.21.15.92:0/3628282742 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6828 (secure 0 0 0) 0x7f8898058e30 con 0x7f88a80c6470 2024-03-19T02:55:00.688 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:00.688 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:54:59.695954+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:55:00.689 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 7 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 msgr2=0x7f888806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 0x7f888806e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8894034000 tx=0x7f8894008030 comp rx=0 tx=0).stop 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 msgr2=0x7f88a8147210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f889800e9b0 tx=0x7f889800ee70 comp rx=0 tx=0).stop 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 shutdown_connections 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88a81593d0 0x7f88a81eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88a8152bf0 0x7f88a814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 --2- 172.21.15.92:0/3628282742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88a80c6470 0x7f88a8147210 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.689+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 >> 172.21.15.92:0/3628282742 conn(0x7f88a80c8d90 msgr2=0x7f88a8147f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:00.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.690+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 shutdown_connections 2024-03-19T02:55:00.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.690+0000 7f88ada00640 1 -- 172.21.15.92:0/3628282742 wait complete. 2024-03-19T02:55:00.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:00.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.913+0000 7efe87800640 1 Processor -- start 2024-03-19T02:55:00.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.913+0000 7efe87800640 1 -- start start 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.914+0000 7efe87800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 0x7efe8006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe87800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 0x7efe800c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe87800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe800c9a80 con 0x7efe800c6470 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe87800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7efe800c6c90 con 0x7efe80152bf0 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe87800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe80159910 con 0x7efe801593d0 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe7fe00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efe800c6470 0x7efe800c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57878/0 (socket says 172.21.15.92:57878) 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe7fe00640 1 -- 172.21.15.92:0/903753686 learned_addr learned my addr 172.21.15.92:0/903753686 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:00.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe85800640 1 --2- 172.21.15.92:0/903753686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 0x7efe800c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe84e00640 1 --2- 172.21.15.92:0/903753686 >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 0x7efe8006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe7f400640 1 -- 172.21.15.92:0/903753686 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039410901 0 0) 0x7efe800c9a80 con 0x7efe800c6470 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe7f400640 1 -- 172.21.15.92:0/903753686 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe68003520 con 0x7efe800c6470 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe85800640 1 -- 172.21.15.92:0/903753686 >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 msgr2=0x7efe8006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.915+0000 7efe85800640 1 --2- 172.21.15.92:0/903753686 >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 0x7efe8006eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.916+0000 7efe85800640 1 -- 172.21.15.92:0/903753686 >> v1:172.21.15.92:6789/0 conn(0x7efe800c6470 legacy=0x7efe800c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.916+0000 7efe85800640 1 -- 172.21.15.92:0/903753686 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe800c8650 con 0x7efe801593d0 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.916+0000 7efe84e00640 1 --2- 172.21.15.92:0/903753686 >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 0x7efe8006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:00.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe85800640 1 --2- 172.21.15.92:0/903753686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 0x7efe800c4690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe78009d90 tx=0x7efe7802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b505bd4e43ae504 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe7f400640 1 -- 172.21.15.92:0/903753686 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78039070 con 0x7efe801593d0 2024-03-19T02:55:00.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe7f400640 1 -- 172.21.15.92:0/903753686 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7802cbd0 con 0x7efe801593d0 2024-03-19T02:55:00.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe7f400640 1 -- 172.21.15.92:0/903753686 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78034440 con 0x7efe801593d0 2024-03-19T02:55:00.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe87800640 1 -- 172.21.15.92:0/903753686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 msgr2=0x7efe800c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.917+0000 7efe87800640 1 --2- 172.21.15.92:0/903753686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 0x7efe800c4690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe78009d90 tx=0x7efe7802be20 comp rx=0 tx=0).stop 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 -- 172.21.15.92:0/903753686 shutdown_connections 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 --2- 172.21.15.92:0/903753686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe801593d0 0x7efe800c4690 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 --2- 172.21.15.92:0/903753686 >> v2:172.21.15.159:3300/0 conn(0x7efe80152bf0 0x7efe8006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 -- 172.21.15.92:0/903753686 >> 172.21.15.92:0/903753686 conn(0x7efe800c8d90 msgr2=0x7efe800c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 -- 172.21.15.92:0/903753686 shutdown_connections 2024-03-19T02:55:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.918+0000 7efe87800640 1 -- 172.21.15.92:0/903753686 wait complete. 2024-03-19T02:55:00.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.919+0000 7efe87800640 1 Processor -- start 2024-03-19T02:55:00.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.919+0000 7efe87800640 1 -- start start 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe80152bf0 0x7efe80159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe801593d0 0x7efe801f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efe8006ea60 con 0x7efe800c6470 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe8006e4a0 con 0x7efe801593d0 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.920+0000 7efe87800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe8006e6f0 con 0x7efe80152bf0 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47790/0 (socket says 172.21.15.92:47790) 2024-03-19T02:55:00.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe80152bf0 0x7efe80159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 -- 172.21.15.92:0/1778466348 learned_addr learned my addr 172.21.15.92:0/1778466348 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe85800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe801593d0 0x7efe801f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 -- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe80152bf0 msgr2=0x7efe80159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe80152bf0 0x7efe80159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 -- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe801593d0 msgr2=0x7efe801f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe801593d0 0x7efe801f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 -- 172.21.15.92:0/1778466348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe801e2fe0 con 0x7efe800c6470 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe84e00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe7400ebe0 tx=0x7efe7400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.921+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74017070 con 0x7efe800c6470 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.922+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7400cc30 con 0x7efe800c6470 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.922+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74013470 con 0x7efe800c6470 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.922+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe801e32c0 con 0x7efe800c6470 2024-03-19T02:55:00.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.922+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe801e3880 con 0x7efe800c6470 2024-03-19T02:55:00.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.923+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe801512f0 con 0x7efe800c6470 2024-03-19T02:55:00.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.924+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe7401a070 con 0x7efe800c6470 2024-03-19T02:55:00.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.924+0000 7efe7cc00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 0x7efe6006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:00.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.925+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efe7408acc0 con 0x7efe800c6470 2024-03-19T02:55:00.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.925+0000 7efe7fe00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 0x7efe6006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:00.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.925+0000 7efe7fe00640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 0x7efe6006e0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe801e5f70 tx=0x7efe6c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:00.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:00.929+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe801512f0 con 0x7efe800c6470 2024-03-19T02:55:01.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.128+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe800684c0 con 0x7efe800c6470 2024-03-19T02:55:01.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.130+0000 7efe7cc00640 1 -- 172.21.15.92:0/1778466348 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7efe74091080 con 0x7efe800c6470 2024-03-19T02:55:01.132 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:01.132 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 msgr2=0x7efe6006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 0x7efe6006e0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe801e5f70 tx=0x7efe6c009040 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 msgr2=0x7efe80144d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe7400ebe0 tx=0x7efe7400c520 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 shutdown_connections 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe6006bc00 0x7efe6006e0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe801593d0 0x7efe801f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe80152bf0 0x7efe80159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 --2- 172.21.15.92:0/1778466348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe800c6470 0x7efe80144d50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 >> 172.21.15.92:0/1778466348 conn(0x7efe800c8d90 msgr2=0x7efe800c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:01.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.133+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 shutdown_connections 2024-03-19T02:55:01.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.134+0000 7efe87800640 1 -- 172.21.15.92:0/1778466348 wait complete. 2024-03-19T02:55:01.153 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T02:55:01.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:01.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.343+0000 7f6be1000640 1 Processor -- start 2024-03-19T02:55:01.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.343+0000 7f6be1000640 1 -- start start 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.344+0000 7f6be1000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bdc151ce0 0x7f6bdc158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.344+0000 7f6be1000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6bdc0c7da0 0x7f6bdc0c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.344+0000 7f6be1000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bdc0c3b80 con 0x7f6bdc151900 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.344+0000 7f6be1000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6bdc158860 con 0x7f6bdc151ce0 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.344+0000 7f6be1000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6bdc072e70 con 0x7f6bdc0c7da0 2024-03-19T02:55:01.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd1400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bdc151ce0 0x7f6bdc158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd9e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6bdc151900 0x7f6bdc0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57888/0 (socket says 172.21.15.92:57888) 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd9e00640 1 -- 172.21.15.92:0/2731033133 learned_addr learned my addr 172.21.15.92:0/2731033133 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bda800640 1 --2- 172.21.15.92:0/2731033133 >> v2:172.21.15.159:3300/0 conn(0x7f6bdc0c7da0 0x7f6bdc0c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd1400640 1 -- 172.21.15.92:0/2731033133 >> v2:172.21.15.159:3300/0 conn(0x7f6bdc0c7da0 msgr2=0x7f6bdc0c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd1400640 1 --2- 172.21.15.92:0/2731033133 >> v2:172.21.15.159:3300/0 conn(0x7f6bdc0c7da0 0x7f6bdc0c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd1400640 1 -- 172.21.15.92:0/2731033133 >> v1:172.21.15.92:6789/0 conn(0x7f6bdc151900 legacy=0x7f6bdc0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.345+0000 7f6bd1400640 1 -- 172.21.15.92:0/2731033133 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc073150 con 0x7f6bdc151ce0 2024-03-19T02:55:01.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.346+0000 7f6bd1400640 1 --2- 172.21.15.92:0/2731033133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc158320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc073700 tx=0x7f6bc402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d354f5c991df21 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.346+0000 7f6bd9400640 1 -- 172.21.15.92:0/2731033133 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4034070 con 0x7f6bdc151ce0 2024-03-19T02:55:01.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.346+0000 7f6bd9400640 1 -- 172.21.15.92:0/2731033133 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bc4030610 con 0x7f6bdc151ce0 2024-03-19T02:55:01.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.346+0000 7f6bd9400640 1 -- 172.21.15.92:0/2731033133 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40309e0 con 0x7f6bdc151ce0 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 -- 172.21.15.92:0/2731033133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 msgr2=0x7f6bdc158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 --2- 172.21.15.92:0/2731033133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc158320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc073700 tx=0x7f6bc402b960 comp rx=0 tx=0).stop 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 -- 172.21.15.92:0/2731033133 shutdown_connections 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 --2- 172.21.15.92:0/2731033133 >> v2:172.21.15.159:3300/0 conn(0x7f6bdc0c7da0 0x7f6bdc0c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 --2- 172.21.15.92:0/2731033133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc158320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 -- 172.21.15.92:0/2731033133 >> 172.21.15.92:0/2731033133 conn(0x7f6bdc0c8d90 msgr2=0x7f6bdc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 -- 172.21.15.92:0/2731033133 shutdown_connections 2024-03-19T02:55:01.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.347+0000 7f6be1000640 1 -- 172.21.15.92:0/2731033133 wait complete. 2024-03-19T02:55:01.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.349+0000 7f6be1000640 1 Processor -- start 2024-03-19T02:55:01.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.349+0000 7f6be1000640 1 -- start start 2024-03-19T02:55:01.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bdc0c7da0 0x7f6bdc14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bdc151900 0x7f6bdc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc0c9af0 con 0x7f6bdc151ce0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc071eb0 con 0x7f6bdc151900 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6be1000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6bdc0c9c70 con 0x7f6bdc0c7da0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bda800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bda800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47812/0 (socket says 172.21.15.92:47812) 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bda800640 1 -- 172.21.15.92:0/2402948451 learned_addr learned my addr 172.21.15.92:0/2402948451 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bd9e00640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bdc0c7da0 0x7f6bdc14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bd1400640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bdc151900 0x7f6bdc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bda800640 1 -- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bdc0c7da0 msgr2=0x7f6bdc14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.350+0000 7f6bda800640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bdc0c7da0 0x7f6bdc14fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bda800640 1 -- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bdc151900 msgr2=0x7f6bdc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bda800640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bdc151900 0x7f6bdc14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bda800640 1 -- 172.21.15.92:0/2402948451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc155040 con 0x7f6bdc151ce0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bda800640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009d30 tx=0x7f6bd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4018020 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd4014480 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc150910 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4014870 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.351+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc1f2b70 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.352+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bdc0684c0 con 0x7f6bdc151ce0 2024-03-19T02:55:01.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.353+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6bd4005d10 con 0x7f6bdc151ce0 2024-03-19T02:55:01.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.354+0000 7f6bd2a00640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 0x7f6bb806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.354+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6bd408b2f0 con 0x7f6bdc151ce0 2024-03-19T02:55:01.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.354+0000 7f6bd9e00640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 0x7f6bb806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.355+0000 7f6bd9e00640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 0x7f6bb806e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007fd0 tx=0x7f6bcc007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.358+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6bd4022070 con 0x7f6bdc151ce0 2024-03-19T02:55:01.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.537+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6bdc146c50 con 0x7f6bb806bbd0 2024-03-19T02:55:01.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.544+0000 7f6bd2a00640 1 -- 172.21.15.92:0/2402948451 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243605 (secure 0 0 0) 0x7f6bdc146c50 con 0x7f6bb806bbd0 2024-03-19T02:55:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:01.551 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 msgr2=0x7f6bb806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 0x7f6bb806e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007fd0 tx=0x7f6bcc007e80 comp rx=0 tx=0).stop 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 msgr2=0x7f6bdc14f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009d30 tx=0x7f6bd400c490 comp rx=0 tx=0).stop 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 shutdown_connections 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6bb806bbd0 0x7f6bb806e090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bdc151ce0 0x7f6bdc14f510 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bdc151900 0x7f6bdc14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 --2- 172.21.15.92:0/2402948451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bdc0c7da0 0x7f6bdc14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 >> 172.21.15.92:0/2402948451 conn(0x7f6bdc0c8d90 msgr2=0x7f6bdc074740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:01.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 shutdown_connections 2024-03-19T02:55:01.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.550+0000 7f6be1000640 1 -- 172.21.15.92:0/2402948451 wait complete. 2024-03-19T02:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-19T02:54:59.836439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":136,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217712,"kb_used_data":1452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757008,"statfs":{"total":773094113280,"available":772871176192,"internally_reserved":0,"allocated":1486848,"data_stored":1600141,"data_compressed":64666,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.661954"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.174282+0000","last_change":"2024-03-19T02:54:55.174282+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.174282+0000","last_undegraded":"2024-03-19T02:54:55.174282+0000","last_fullsized":"2024-03-19T02:54:55.174282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.170664+0000","last_change":"2024-03-19T02:54:55.170664+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.170664+0000","last_undegraded":"2024-03-19T02:54:55.170664+0000","last_fullsized":"2024-03-19T02:54:55.170664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.174309+0000","last_change":"2024-03-19T02:54:55.174309+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.174309+0000","last_undegraded":"2024-03-19T02:54:55.174309+0000","last_fullsized":"2024-03-19T02:54:55.174309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.170526+0000","last_change":"2024-03-19T02:54:55.170526+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.170526+0000","last_undegraded":"2024-03-19T02:54:55.170526+0000","last_fullsized":"2024-03-19T02:54:55.170526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173724+0000","last_change":"2024-03-19T02:54:55.173724+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173724+0000","last_undegraded":"2024-03-19T02:54:55.173724+0000","last_fullsized":"2024-03-19T02:54:55.173724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173807+0000","last_change":"2024-03-19T02:54:55.173807+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173807+0000","last_undegraded":"2024-03-19T02:54:55.173807+0000","last_fullsized":"2024-03-19T02:54:55.173807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.178740+0000","last_change":"2024-03-19T02:54:55.178740+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.178740+0000","last_undegraded":"2024-03-19T02:54:55.178740+0000","last_fullsized":"2024-03-19T02:54:55.178740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.177351+0000","last_change":"2024-03-19T02:54:55.177351+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.177351+0000","last_undegraded":"2024-03-19T02:54:55.177351+0000","last_fullsized":"2024-03-19T02:54:55.177351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.172599+0000","last_change":"2024-03-19T02:54:55.172599+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.172599+0000","last_undegraded":"2024-03-19T02:54:55.172599+0000","last_fullsized":"2024-03-19T02:54:55.172599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.170145+0000","last_change":"2024-03-19T02:54:55.170145+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.170145+0000","last_undegraded":"2024-03-19T02:54:55.170145+0000","last_fullsized":"2024-03-19T02:54:55.170145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.170139+0000","last_change":"2024-03-19T02:54:55.170139+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.170139+0000","last_undegraded":"2024-03-19T02:54:55.170139+0000","last_fullsized":"2024-03-19T02:54:55.170139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173898+0000","last_change":"2024-03-19T02:54:55.173898+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173898+0000","last_undegraded":"2024-03-19T02:54:55.173898+0000","last_fullsized":"2024-03-19T02:54:55.173898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.177549+0000","last_change":"2024-03-19T02:54:55.177549+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.177549+0000","last_undegraded":"2024-03-19T02:54:55.177549+0000","last_fullsized":"2024-03-19T02:54:55.177549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.170039+0000","last_change":"2024-03-19T02:54:55.170039+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.170039+0000","last_undegraded":"2024-03-19T02:54:55.170039+0000","last_fullsized":"2024-03-19T02:54:55.170039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.169461+0000","last_change":"2024-03-19T02:54:55.169461+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.169461+0000","last_undegraded":"2024-03-19T02:54:55.169461+0000","last_fullsized":"2024-03-19T02:54:55.169461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.178354+0000","last_change":"2024-03-19T02:54:55.178354+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.178354+0000","last_undegraded":"2024-03-19T02:54:55.178354+0000","last_fullsized":"2024-03-19T02:54:55.178354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.169543+0000","last_change":"2024-03-19T02:54:55.169543+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.169543+0000","last_undegraded":"2024-03-19T02:54:55.169543+0000","last_fullsized":"2024-03-19T02:54:55.169543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.178639+0000","last_change":"2024-03-19T02:54:55.178639+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.178639+0000","last_undegraded":"2024-03-19T02:54:55.178639+0000","last_fullsized":"2024-03-19T02:54:55.178639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.171556+0000","last_change":"2024-03-19T02:54:55.171556+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.171556+0000","last_undegraded":"2024-03-19T02:54:55.171556+0000","last_fullsized":"2024-03-19T02:54:55.171556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.169103+0000","last_change":"2024-03-19T02:54:54.169103+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.169103+0000","last_undegraded":"2024-03-19T02:54:54.169103+0000","last_fullsized":"2024-03-19T02:54:54.169103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.185188+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:55.185188+0000","last_peered":"2024-03-19T02:54:55.185188+0000","last_clean":"2024-03-19T02:54:55.185188+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:55.185188+0000","last_undegraded":"2024-03-19T02:54:55.185188+0000","last_fullsized":"2024-03-19T02:54:55.185188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:34.765772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.170135+0000","last_change":"2024-03-19T02:54:54.170135+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.170135+0000","last_undegraded":"2024-03-19T02:54:54.170135+0000","last_fullsized":"2024-03-19T02:54:54.170135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.180294+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:55.180294+0000","last_peered":"2024-03-19T02:54:55.180294+0000","last_clean":"2024-03-19T02:54:55.180294+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:55.180294+0000","last_undegraded":"2024-03-19T02:54:55.180294+0000","last_fullsized":"2024-03-19T02:54:55.180294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:30.402778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.184615+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:55.184615+0000","last_peered":"2024-03-19T02:54:55.184615+0000","last_clean":"2024-03-19T02:54:55.184615+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:55.184615+0000","last_undegraded":"2024-03-19T02:54:55.184615+0000","last_fullsized":"2024-03-19T02:54:55.184615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:06:06.566675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.169053+0000","last_change":"2024-03-19T02:54:55.169053+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.169053+0000","last_undegraded":"2024-03-19T02:54:55.169053+0000","last_fullsized":"2024-03-19T02:54:55.169053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.184842+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:55.184842+0000","last_peered":"2024-03-19T02:54:55.184842+0000","last_clean":"2024-03-19T02:54:55.184842+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:55.184842+0000","last_undegraded":"2024-03-19T02:54:55.184842+0000","last_fullsized":"2024-03-19T02:54:55.184842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:09.201479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.181961+0000","last_change":"2024-03-19T02:54:54.181961+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.181961+0000","last_undegraded":"2024-03-19T02:54:54.181961+0000","last_fullsized":"2024-03-19T02:54:54.181961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.172116+0000","last_change":"2024-03-19T02:54:55.172116+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.172116+0000","last_undegraded":"2024-03-19T02:54:55.172116+0000","last_fullsized":"2024-03-19T02:54:55.172116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.192807+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:54:55.192807+0000","last_peered":"2024-03-19T02:54:55.192807+0000","last_clean":"2024-03-19T02:54:55.192807+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:54:55.192807+0000","last_undegraded":"2024-03-19T02:54:55.192807+0000","last_fullsized":"2024-03-19T02:54:55.192807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:19.713074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.176282+0000","last_change":"2024-03-19T02:54:55.176282+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.176282+0000","last_undegraded":"2024-03-19T02:54:55.176282+0000","last_fullsized":"2024-03-19T02:54:55.176282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.172327+0000","last_change":"2024-03-19T02:54:54.172327+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.172327+0000","last_undegraded":"2024-03-19T02:54:54.172327+0000","last_fullsized":"2024-03-19T02:54:54.172327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.200737+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:55.200737+0000","last_peered":"2024-03-19T02:54:55.200737+0000","last_clean":"2024-03-19T02:54:55.200737+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:55.200737+0000","last_undegraded":"2024-03-19T02:54:55.200737+0000","last_fullsized":"2024-03-19T02:54:55.200737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:25.093830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173261+0000","last_change":"2024-03-19T02:54:55.173261+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173261+0000","last_undegraded":"2024-03-19T02:54:55.173261+0000","last_fullsized":"2024-03-19T02:54:55.173261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.184694+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:55.184694+0000","last_peered":"2024-03-19T02:54:55.184694+0000","last_clean":"2024-03-19T02:54:55.184694+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:55.184694+0000","last_undegraded":"2024-03-19T02:54:55.184694+0000","last_fullsized":"2024-03-19T02:54:55.184694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:28.193202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.176612+0000","last_change":"2024-03-19T02:54:55.176612+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.176612+0000","last_undegraded":"2024-03-19T02:54:55.176612+0000","last_fullsized":"2024-03-19T02:54:55.176612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.174345+0000","last_change":"2024-03-19T02:54:54.174345+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.174345+0000","last_undegraded":"2024-03-19T02:54:54.174345+0000","last_fullsized":"2024-03-19T02:54:54.174345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173139+0000","last_change":"2024-03-19T02:54:55.173139+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173139+0000","last_undegraded":"2024-03-19T02:54:55.173139+0000","last_fullsized":"2024-03-19T02:54:55.173139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.199241+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:55.199241+0000","last_peered":"2024-03-19T02:54:55.199241+0000","last_clean":"2024-03-19T02:54:55.199241+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:55.199241+0000","last_undegraded":"2024-03-19T02:54:55.199241+0000","last_fullsized":"2024-03-19T02:54:55.199241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:14.215107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173253+0000","last_change":"2024-03-19T02:54:55.173253+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173253+0000","last_undegraded":"2024-03-19T02:54:55.173253+0000","last_fullsized":"2024-03-19T02:54:55.173253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.173001+0000","last_change":"2024-03-19T02:54:54.173001+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.173001+0000","last_undegraded":"2024-03-19T02:54:54.173001+0000","last_fullsized":"2024-03-19T02:54:54.173001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.173302+0000","last_change":"2024-03-19T02:54:55.173302+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.173302+0000","last_undegraded":"2024-03-19T02:54:55.173302+0000","last_fullsized":"2024-03-19T02:54:55.173302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.177979+0000","last_change":"2024-03-19T02:54:54.177979+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.177979+0000","last_undegraded":"2024-03-19T02:54:54.177979+0000","last_fullsized":"2024-03-19T02:54:54.177979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.168777+0000","last_change":"2024-03-19T02:54:55.168777+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.168777+0000","last_undegraded":"2024-03-19T02:54:55.168777+0000","last_fullsized":"2024-03-19T02:54:55.168777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.172782+0000","last_change":"2024-03-19T02:54:54.172782+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.172782+0000","last_undegraded":"2024-03-19T02:54:54.172782+0000","last_fullsized":"2024-03-19T02:54:54.172782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.175825+0000","last_change":"2024-03-19T02:54:55.175825+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.175825+0000","last_undegraded":"2024-03-19T02:54:55.175825+0000","last_fullsized":"2024-03-19T02:54:55.175825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.778103+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:55.778103+0000","last_peered":"2024-03-19T02:54:55.778103+0000","last_clean":"2024-03-19T02:54:55.778103+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:55.778103+0000","last_undegraded":"2024-03-19T02:54:55.778103+0000","last_fullsized":"2024-03-19T02:54:55.778103+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.183987+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:55.183987+0000","last_peered":"2024-03-19T02:54:55.183987+0000","last_clean":"2024-03-19T02:54:55.183987+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:55.183987+0000","last_undegraded":"2024-03-19T02:54:55.183987+0000","last_fullsized":"2024-03-19T02:54:55.183987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:58.790289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.199197+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:55.199197+0000","last_peered":"2024-03-19T02:54:55.199197+0000","last_clean":"2024-03-19T02:54:55.199197+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:55.199197+0000","last_undegraded":"2024-03-19T02:54:55.199197+0000","last_fullsized":"2024-03-19T02:54:55.199197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:12.188320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.172787+0000","last_change":"2024-03-19T02:54:54.172787+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.172787+0000","last_undegraded":"2024-03-19T02:54:54.172787+0000","last_fullsized":"2024-03-19T02:54:54.172787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.201503+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:55.201503+0000","last_peered":"2024-03-19T02:54:55.201503+0000","last_clean":"2024-03-19T02:54:55.201503+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:55.201503+0000","last_undegraded":"2024-03-19T02:54:55.201503+0000","last_fullsized":"2024-03-19T02:54:55.201503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:49.370534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.171496+0000","last_change":"2024-03-19T02:54:55.171496+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.171496+0000","last_undegraded":"2024-03-19T02:54:55.171496+0000","last_fullsized":"2024-03-19T02:54:55.171496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.177902+0000","last_change":"2024-03-19T02:54:54.177902+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.177902+0000","last_undegraded":"2024-03-19T02:54:54.177902+0000","last_fullsized":"2024-03-19T02:54:54.177902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.168790+0000","last_change":"2024-03-19T02:54:55.168790+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.168790+0000","last_undegraded":"2024-03-19T02:54:55.168790+0000","last_fullsized":"2024-03-19T02:54:55.168790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.204784+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:55.204784+0000","last_peered":"2024-03-19T02:54:55.204784+0000","last_clean":"2024-03-19T02:54:55.204784+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:55.204784+0000","last_undegraded":"2024-03-19T02:54:55.204784+0000","last_fullsized":"2024-03-19T02:54:55.204784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:58.271141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.177115+0000","last_change":"2024-03-19T02:54:54.177115+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.177115+0000","last_undegraded":"2024-03-19T02:54:54.177115+0000","last_fullsized":"2024-03-19T02:54:54.177115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.184261+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:54:55.184261+0000","last_peered":"2024-03-19T02:54:55.184261+0000","last_clean":"2024-03-19T02:54:55.184261+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:54:55.184261+0000","last_undegraded":"2024-03-19T02:54:55.184261+0000","last_fullsized":"2024-03-19T02:54:55.184261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:48.928113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.200583+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:55.200583+0000","last_peered":"2024-03-19T02:54:55.200583+0000","last_clean":"2024-03-19T02:54:55.200583+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:55.200583+0000","last_undegraded":"2024-03-19T02:54:55.200583+0000","last_fullsized":"2024-03-19T02:54:55.200583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:52.477584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.171679+0000","last_change":"2024-03-19T02:54:55.171679+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.171679+0000","last_undegraded":"2024-03-19T02:54:55.171679+0000","last_fullsized":"2024-03-19T02:54:55.171679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.176532+0000","last_change":"2024-03-19T02:54:54.176532+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.176532+0000","last_undegraded":"2024-03-19T02:54:54.176532+0000","last_fullsized":"2024-03-19T02:54:54.176532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.177482+0000","last_change":"2024-03-19T02:54:54.177482+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.177482+0000","last_undegraded":"2024-03-19T02:54:54.177482+0000","last_fullsized":"2024-03-19T02:54:54.177482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.183372+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:54:55.183372+0000","last_peered":"2024-03-19T02:54:55.183372+0000","last_clean":"2024-03-19T02:54:55.183372+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:54:55.183372+0000","last_undegraded":"2024-03-19T02:54:55.183372+0000","last_fullsized":"2024-03-19T02:54:55.183372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:08.841310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.176428+0000","last_change":"2024-03-19T02:54:55.176428+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.176428+0000","last_undegraded":"2024-03-19T02:54:55.176428+0000","last_fullsized":"2024-03-19T02:54:55.176428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.169407+0000","last_change":"2024-03-19T02:54:55.169407+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.169407+0000","last_undegraded":"2024-03-19T02:54:55.169407+0000","last_fullsized":"2024-03-19T02:54:55.169407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.183326+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:54:55.183326+0000","last_peered":"2024-03-19T02:54:55.183326+0000","last_clean":"2024-03-19T02:54:55.183326+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:54:55.183326+0000","last_undegraded":"2024-03-19T02:54:55.183326+0000","last_fullsized":"2024-03-19T02:54:55.183326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.525569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-19T02:54:55.169481+0000","last_change":"2024-03-19T02:54:55.169481+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.169481+0000","last_undegraded":"2024-03-19T02:54:55.169481+0000","last_fullsized":"2024-03-19T02:54:55.169481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T02:54:55.184098+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:54:55.184098+0000","last_peered":"2024-03-19T02:54:55.184098+0000","last_clean":"2024-03-19T02:54:55.184098+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:54:55.184098+0000","last_undegraded":"2024-03-19T02:54:55.184098+0000","last_fullsized":"2024-03-19T02:54:55.184098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:23.712690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.168311+0000","last_change":"2024-03-19T02:54:54.168311+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.168311+0000","last_undegraded":"2024-03-19T02:54:54.168311+0000","last_fullsized":"2024-03-19T02:54:54.168311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T02:54:55.159487+0000","last_change":"2024-03-19T02:54:55.159487+0000","last_active":"2024-03-19T02:54:55.159487+0000","last_peered":"2024-03-19T02:54:55.159487+0000","last_clean":"2024-03-19T02:54:55.159487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:55.159487+0000","last_undegraded":"2024-03-19T02:54:55.159487+0000","last_fullsized":"2024-03-19T02:54:55.159487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-19T02:54:54.172901+0000","last_change":"2024-03-19T02:54:54.172901+0000","last_active":"2024-03-19T02:54:54.149570+0000","last_peered":"2024-03-19T02:54:54.149570+0000","last_clean":"2024-03-19T02:54:54.149570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T02:54:54.172901+0000","last_undegraded":"2024-03-19T02:54:54.172901+0000","last_fullsized":"2024-03-19T02:54:54.172901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":70,"acting":70,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64729,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":245760,"data_stored":648227,"data_compressed":32333,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":233472,"data_stored":641952,"data_compressed":32333,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32333,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:01.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:01.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.816+0000 7f7d65e00640 1 Processor -- start 2024-03-19T02:55:01.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.816+0000 7f7d65e00640 1 -- start start 2024-03-19T02:55:01.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d65e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d65e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7d601593d0 0x7f7d600c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d65e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d600c9a80 con 0x7f7d600c6470 2024-03-19T02:55:01.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d65e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7d600c6c90 con 0x7f7d601593d0 2024-03-19T02:55:01.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d65e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7d60159910 con 0x7f7d60152bf0 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d5f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7d601593d0 0x7f7d600c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6006eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37550/0 (socket says 172.21.15.92:37550) 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.818+0000 7f7d5ea00640 1 -- 172.21.15.92:0/1634482437 learned_addr learned my addr 172.21.15.92:0/1634482437 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.819+0000 7f7d5f400640 1 -- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 msgr2=0x7f7d6006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:01.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.819+0000 7f7d5f400640 1 --2- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6006eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.819+0000 7f7d5f400640 1 -- 172.21.15.92:0/1634482437 >> v1:172.21.15.159:6789/0 conn(0x7f7d600c6470 legacy=0x7f7d600c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.819+0000 7f7d5f400640 1 -- 172.21.15.92:0/1634482437 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d600c8650 con 0x7f7d601593d0 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d5f400640 1 --2- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d601593d0 0x7f7d600c4690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d580097a0 tx=0x7f7d5802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4863c366f9d581a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d5e000640 1 -- 172.21.15.92:0/1634482437 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58031780 con 0x7f7d601593d0 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d5e000640 1 -- 172.21.15.92:0/1634482437 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58037070 con 0x7f7d601593d0 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d5e000640 1 -- 172.21.15.92:0/1634482437 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58031b40 con 0x7f7d601593d0 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d65e00640 1 -- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d601593d0 msgr2=0x7f7d600c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.820+0000 7f7d65e00640 1 --2- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d601593d0 0x7f7d600c4690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d580097a0 tx=0x7f7d5802b820 comp rx=0 tx=0).stop 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 -- 172.21.15.92:0/1634482437 shutdown_connections 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 --2- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d601593d0 0x7f7d600c4690 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 --2- 172.21.15.92:0/1634482437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 -- 172.21.15.92:0/1634482437 >> 172.21.15.92:0/1634482437 conn(0x7f7d600c8d90 msgr2=0x7f7d600c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:01.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 -- 172.21.15.92:0/1634482437 shutdown_connections 2024-03-19T02:55:01.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.821+0000 7f7d65e00640 1 -- 172.21.15.92:0/1634482437 wait complete. 2024-03-19T02:55:01.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.822+0000 7f7d65e00640 1 Processor -- start 2024-03-19T02:55:01.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.822+0000 7f7d65e00640 1 -- start start 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6014e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d601593d0 0x7f7d60144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6006ea60 con 0x7f7d601593d0 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6006e4a0 con 0x7f7d600c6470 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d65e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6006e6f0 con 0x7f7d60152bf0 2024-03-19T02:55:01.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.823+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d601593d0 0x7f7d60144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60706/0 (socket says 172.21.15.92:60706) 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 -- 172.21.15.92:0/3550824470 learned_addr learned my addr 172.21.15.92:0/3550824470 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d56000640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6014e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 -- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 msgr2=0x7f7d6014e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6014e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 -- 172.21.15.92:0/3550824470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d601593d0 msgr2=0x7f7d60144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d601593d0 0x7f7d60144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 -- 172.21.15.92:0/3550824470 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6014dba0 con 0x7f7d600c6470 2024-03-19T02:55:01.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d5ea00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009d90 tx=0x7f7d5000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5000cda0 con 0x7f7d600c6470 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d50019070 con 0x7f7d600c6470 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50014680 con 0x7f7d600c6470 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.824+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d60072c50 con 0x7f7d600c6470 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.825+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d60156570 con 0x7f7d600c6470 2024-03-19T02:55:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.826+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d600684c0 con 0x7f7d600c6470 2024-03-19T02:55:01.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.827+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7d50005d10 con 0x7f7d600c6470 2024-03-19T02:55:01.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.827+0000 7f7d57e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 0x7f7d3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:01.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.827+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7d5008aef0 con 0x7f7d600c6470 2024-03-19T02:55:01.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.828+0000 7f7d56000640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 0x7f7d3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:01.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.828+0000 7f7d56000640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 0x7f7d3c06e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7d60148e60 tx=0x7f7d48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:01.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:01.832+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7d50010040 con 0x7f7d600c6470 2024-03-19T02:55:02.014 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.012+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d60156040 con 0x7f7d3c06bbd0 2024-03-19T02:55:02.021 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.020+0000 7f7d57e00640 1 -- 172.21.15.92:0/3550824470 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251927 (secure 0 0 0) 0x7f7d60156040 con 0x7f7d3c06bbd0 2024-03-19T02:55:02.024 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:02.027 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 msgr2=0x7f7d3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 0x7f7d3c06e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7d60148e60 tx=0x7f7d48009040 comp rx=0 tx=0).stop 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 msgr2=0x7f7d6014ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009d90 tx=0x7f7d5000c490 comp rx=0 tx=0).stop 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 shutdown_connections 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7d3c06bbd0 0x7f7d3c06e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d601593d0 0x7f7d60144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d60152bf0 0x7f7d6014e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 --2- 172.21.15.92:0/3550824470 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d600c6470 0x7f7d6014ec90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 >> 172.21.15.92:0/3550824470 conn(0x7f7d600c8d90 msgr2=0x7f7d60151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:02.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 shutdown_connections 2024-03-19T02:55:02.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.027+0000 7f7d65e00640 1 -- 172.21.15.92:0/3550824470 wait complete. 2024-03-19T02:55:02.045 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-19T02:55:01.836850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217932,"kb_used_data":1728,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756788,"statfs":{"total":773094113280,"available":772870950912,"internally_reserved":0,"allocated":1769472,"data_stored":1748208,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.662365"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380755+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380755+0000","last_peered":"2024-03-19T02:54:57.380755+0000","last_clean":"2024-03-19T02:54:57.380755+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380755+0000","last_undegraded":"2024-03-19T02:54:57.380755+0000","last_fullsized":"2024-03-19T02:54:57.380755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081515+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081515+0000","last_peered":"2024-03-19T02:54:59.081515+0000","last_clean":"2024-03-19T02:54:59.081515+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081515+0000","last_undegraded":"2024-03-19T02:54:59.081515+0000","last_fullsized":"2024-03-19T02:54:59.081515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201424+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:54:57.201424+0000","last_peered":"2024-03-19T02:54:57.201424+0000","last_clean":"2024-03-19T02:54:57.201424+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:54:57.201424+0000","last_undegraded":"2024-03-19T02:54:57.201424+0000","last_fullsized":"2024-03-19T02:54:57.201424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081605+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:54:59.081605+0000","last_peered":"2024-03-19T02:54:59.081605+0000","last_clean":"2024-03-19T02:54:59.081605+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:54:59.081605+0000","last_undegraded":"2024-03-19T02:54:59.081605+0000","last_fullsized":"2024-03-19T02:54:59.081605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084161+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:54:59.084161+0000","last_peered":"2024-03-19T02:54:59.084161+0000","last_clean":"2024-03-19T02:54:59.084161+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:54:59.084161+0000","last_undegraded":"2024-03-19T02:54:59.084161+0000","last_fullsized":"2024-03-19T02:54:59.084161+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196560+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:54:57.196560+0000","last_peered":"2024-03-19T02:54:57.196560+0000","last_clean":"2024-03-19T02:54:57.196560+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:54:57.196560+0000","last_undegraded":"2024-03-19T02:54:57.196560+0000","last_fullsized":"2024-03-19T02:54:57.196560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197906+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197906+0000","last_peered":"2024-03-19T02:54:57.197906+0000","last_clean":"2024-03-19T02:54:57.197906+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197906+0000","last_undegraded":"2024-03-19T02:54:57.197906+0000","last_fullsized":"2024-03-19T02:54:57.197906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197643+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:54:57.197643+0000","last_peered":"2024-03-19T02:54:57.197643+0000","last_clean":"2024-03-19T02:54:57.197643+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:54:57.197643+0000","last_undegraded":"2024-03-19T02:54:57.197643+0000","last_fullsized":"2024-03-19T02:54:57.197643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201597+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201597+0000","last_peered":"2024-03-19T02:54:57.201597+0000","last_clean":"2024-03-19T02:54:57.201597+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201597+0000","last_undegraded":"2024-03-19T02:54:57.201597+0000","last_fullsized":"2024-03-19T02:54:57.201597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201716+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201716+0000","last_peered":"2024-03-19T02:54:57.201716+0000","last_clean":"2024-03-19T02:54:57.201716+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201716+0000","last_undegraded":"2024-03-19T02:54:57.201716+0000","last_fullsized":"2024-03-19T02:54:57.201716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081820+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:54:59.081820+0000","last_peered":"2024-03-19T02:54:59.081820+0000","last_clean":"2024-03-19T02:54:59.081820+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:54:59.081820+0000","last_undegraded":"2024-03-19T02:54:59.081820+0000","last_fullsized":"2024-03-19T02:54:59.081820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082253+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:54:59.082253+0000","last_peered":"2024-03-19T02:54:59.082253+0000","last_clean":"2024-03-19T02:54:59.082253+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:54:59.082253+0000","last_undegraded":"2024-03-19T02:54:59.082253+0000","last_fullsized":"2024-03-19T02:54:59.082253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081970+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:54:59.081970+0000","last_peered":"2024-03-19T02:54:59.081970+0000","last_clean":"2024-03-19T02:54:59.081970+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:54:59.081970+0000","last_undegraded":"2024-03-19T02:54:59.081970+0000","last_fullsized":"2024-03-19T02:54:59.081970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082243+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:54:59.082243+0000","last_peered":"2024-03-19T02:54:59.082243+0000","last_clean":"2024-03-19T02:54:59.082243+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:54:59.082243+0000","last_undegraded":"2024-03-19T02:54:59.082243+0000","last_fullsized":"2024-03-19T02:54:59.082243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381432+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381432+0000","last_peered":"2024-03-19T02:54:57.381432+0000","last_clean":"2024-03-19T02:54:57.381432+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381432+0000","last_undegraded":"2024-03-19T02:54:57.381432+0000","last_fullsized":"2024-03-19T02:54:57.381432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084135+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:54:59.084135+0000","last_peered":"2024-03-19T02:54:59.084135+0000","last_clean":"2024-03-19T02:54:59.084135+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:54:59.084135+0000","last_undegraded":"2024-03-19T02:54:59.084135+0000","last_fullsized":"2024-03-19T02:54:59.084135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":661302,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":661302,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:02.049 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T02:55:02.049 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T02:55:02.049 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-19T02:55:02.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-19T02:55:02.050 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-19T02:55:02.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.292+0000 7f829b200640 1 Processor -- start 2024-03-19T02:55:02.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.292+0000 7f829b200640 1 -- start start 2024-03-19T02:55:02.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f829c154de0 0x7f829c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829b200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f829c0cd1c0 0x7f829c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829b200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f829c0b3290 con 0x7f829c0b34c0 2024-03-19T02:55:02.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829b200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f829c0cdb30 con 0x7f829c154de0 2024-03-19T02:55:02.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829b200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f829c0cccc0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f8299e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f829c154de0 0x7f829c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f829a800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f829c0cd1c0 0x7f829c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f8293e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f829c0b34c0 0x7f829c06bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46844/0 (socket says 172.21.15.92:46844) 2024-03-19T02:55:02.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.294+0000 7f8293e00640 1 -- 172.21.15.92:0/4272943316 learned_addr learned my addr 172.21.15.92:0/4272943316 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f8299400640 1 -- 172.21.15.92:0/4272943316 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863579159 0 0) 0x7f829c0b3290 con 0x7f829c0b34c0 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f8299400640 1 -- 172.21.15.92:0/4272943316 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8280003520 con 0x7f829c0b34c0 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f829a800640 1 -- 172.21.15.92:0/4272943316 >> v2:172.21.15.92:3300/0 conn(0x7f829c154de0 msgr2=0x7f829c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f829a800640 1 --2- 172.21.15.92:0/4272943316 >> v2:172.21.15.92:3300/0 conn(0x7f829c154de0 0x7f829c0cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f829a800640 1 -- 172.21.15.92:0/4272943316 >> v1:172.21.15.159:6789/0 conn(0x7f829c0b34c0 legacy=0x7f829c06bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.295+0000 7f829a800640 1 -- 172.21.15.92:0/4272943316 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c0b2cf0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.296+0000 7f829a800640 1 --2- 172.21.15.92:0/4272943316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c0cd1c0 0x7f829c0cd5f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f829c0cbcd0 tx=0x7f829402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a9ed5bef95b8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.296+0000 7f8299400640 1 -- 172.21.15.92:0/4272943316 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294039070 con 0x7f829c0cd1c0 2024-03-19T02:55:02.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.296+0000 7f8299400640 1 -- 172.21.15.92:0/4272943316 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f829402cc80 con 0x7f829c0cd1c0 2024-03-19T02:55:02.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.296+0000 7f8299400640 1 -- 172.21.15.92:0/4272943316 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294034660 con 0x7f829c0cd1c0 2024-03-19T02:55:02.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 -- 172.21.15.92:0/4272943316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c0cd1c0 msgr2=0x7f829c0cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 --2- 172.21.15.92:0/4272943316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c0cd1c0 0x7f829c0cd5f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f829c0cbcd0 tx=0x7f829402bed0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 -- 172.21.15.92:0/4272943316 shutdown_connections 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 --2- 172.21.15.92:0/4272943316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c0cd1c0 0x7f829c0cd5f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 --2- 172.21.15.92:0/4272943316 >> v2:172.21.15.92:3300/0 conn(0x7f829c154de0 0x7f829c0cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 -- 172.21.15.92:0/4272943316 >> 172.21.15.92:0/4272943316 conn(0x7f829c067130 msgr2=0x7f829c067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 -- 172.21.15.92:0/4272943316 shutdown_connections 2024-03-19T02:55:02.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.297+0000 7f829b200640 1 -- 172.21.15.92:0/4272943316 wait complete. 2024-03-19T02:55:02.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.299+0000 7f829b200640 1 Processor -- start 2024-03-19T02:55:02.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.299+0000 7f829b200640 1 -- start start 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f829c0b34c0 0x7f829c14d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c154de0 0x7f829c14e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f829c15e1d0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f829c0b2fb0 con 0x7f829c154de0 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f829b200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f829c15e4b0 con 0x7f829c0b34c0 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.300+0000 7f8293e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47838/0 (socket says 172.21.15.92:47838) 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 -- 172.21.15.92:0/4214331655 learned_addr learned my addr 172.21.15.92:0/4214331655 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f829a800640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c154de0 0x7f829c14e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8299e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f829c0b34c0 0x7f829c14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.303 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 -- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f829c0b34c0 msgr2=0x7f829c14d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f829c0b34c0 0x7f829c14d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 -- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c154de0 msgr2=0x7f829c14e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c154de0 0x7f829c14e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 -- 172.21.15.92:0/4214331655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c15d9f0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8293e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f828400c430 tx=0x7f828400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284017070 con 0x7f829c0cd1c0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284005ce0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.301+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82840072f0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.302+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c15dcd0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.302+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c1f3ad0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.303+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829c04f580 con 0x7f829c0cd1c0 2024-03-19T02:55:02.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.304+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f828401a070 con 0x7f829c0cd1c0 2024-03-19T02:55:02.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.304+0000 7f8292000640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 0x7f827806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.305+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8284089ef0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.305+0000 7f8299e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 0x7f827806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.305+0000 7f8299e00640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 0x7f827806e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f828c009aa0 tx=0x7f828c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.310 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.309+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8284058300 con 0x7f829c0cd1c0 2024-03-19T02:55:02.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.538+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f829c0cbd20 con 0x7f829c0cd1c0 2024-03-19T02:55:02.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.543+0000 7f8292000640 1 -- 172.21.15.92:0/4214331655 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172583 (secure 0 0 0) 0x7f828405c3a0 con 0x7f829c0cd1c0 2024-03-19T02:55:02.546 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 msgr2=0x7f827806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 0x7f827806e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f828c009aa0 tx=0x7f828c009190 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 msgr2=0x7f829c14de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f828400c430 tx=0x7f828400c8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 shutdown_connections 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f827806bc00 0x7f827806e0c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f829c154de0 0x7f829c14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f829c0cd1c0 0x7f829c14de60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 --2- 172.21.15.92:0/4214331655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f829c0b34c0 0x7f829c14d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.547+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 >> 172.21.15.92:0/4214331655 conn(0x7f829c067130 msgr2=0x7f829c0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:02.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.548+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 shutdown_connections 2024-03-19T02:55:02.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.548+0000 7f829b200640 1 -- 172.21.15.92:0/4214331655 wait complete. 2024-03-19T02:55:02.565 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":697081936},{"type":"v1","addr":"172.21.15.159:6833","nonce":697081936}]},"active_addr":"172.21.15.159:6833/697081936","active_change":"2024-03-19T02:54:43.823756+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":2661958149}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":2162252284}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3226267078}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3743646609}]}]} 2024-03-19T02:55:02.570 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-19T02:55:02.570 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-19T02:55:02.571 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:02.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.770+0000 7f4180000640 1 Processor -- start 2024-03-19T02:55:02.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.770+0000 7f4180000640 1 -- start start 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.771+0000 7f4180000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f41781550b0 0x7f417815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.771+0000 7f4180000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41780caf90 con 0x7f4178067ad0 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.771+0000 7f4180000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417815e280 con 0x7f4178154cd0 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.771+0000 7f4180000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f417815e520 con 0x7f41781550b0 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f41781550b0 0x7f417815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f41781550b0 0x7f417815ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47844/0 (socket says 172.21.15.92:47844) 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417e000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4178067ad0 0x7f41780cbfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57892/0 (socket says 172.21.15.92:57892) 2024-03-19T02:55:02.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 -- 172.21.15.92:0/936346619 learned_addr learned my addr 172.21.15.92:0/936346619 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:02.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 -- 172.21.15.92:0/936346619 >> v1:172.21.15.159:6789/0 conn(0x7f4178154cd0 legacy=0x7f41780cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 -- 172.21.15.92:0/936346619 >> v1:172.21.15.92:6789/0 conn(0x7f4178067ad0 legacy=0x7f41780cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f4177e00640 1 -- 172.21.15.92:0/936346619 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456265525 0 0) 0x7f417815e280 con 0x7f4178154cd0 2024-03-19T02:55:02.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.772+0000 7f417cc00640 1 -- 172.21.15.92:0/936346619 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417815e800 con 0x7f41781550b0 2024-03-19T02:55:02.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.773+0000 7f417cc00640 1 --2- 172.21.15.92:0/936346619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f417815ab80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f417815e450 tx=0x7f416402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8f9d7b131b8244 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.773+0000 7f4177e00640 1 -- 172.21.15.92:0/936346619 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416402f830 con 0x7f41781550b0 2024-03-19T02:55:02.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.773+0000 7f4177e00640 1 -- 172.21.15.92:0/936346619 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416402f9d0 con 0x7f41781550b0 2024-03-19T02:55:02.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.773+0000 7f4177e00640 1 -- 172.21.15.92:0/936346619 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416402fda0 con 0x7f41781550b0 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 -- 172.21.15.92:0/936346619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 msgr2=0x7f417815ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 --2- 172.21.15.92:0/936346619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f417815ab80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f417815e450 tx=0x7f416402ba10 comp rx=0 tx=0).stop 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 -- 172.21.15.92:0/936346619 shutdown_connections 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 --2- 172.21.15.92:0/936346619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f417815ab80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 -- 172.21.15.92:0/936346619 >> 172.21.15.92:0/936346619 conn(0x7f4178072970 msgr2=0x7f4178066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 -- 172.21.15.92:0/936346619 shutdown_connections 2024-03-19T02:55:02.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.774+0000 7f4180000640 1 -- 172.21.15.92:0/936346619 wait complete. 2024-03-19T02:55:02.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.776+0000 7f4180000640 1 Processor -- start 2024-03-19T02:55:02.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.776+0000 7f4180000640 1 -- start start 2024-03-19T02:55:02.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.777+0000 7f4180000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4178067ad0 0x7f4178141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f4180000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f4180000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f41781f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f4180000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f41780cd5c0 con 0x7f41781550b0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f4180000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f417815edc0 con 0x7f4178067ad0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f4180000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f41780cd740 con 0x7f4178154cd0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417d600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4178067ad0 0x7f4178141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37586/0 (socket says 172.21.15.92:37586) 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 -- 172.21.15.92:0/433991608 learned_addr learned my addr 172.21.15.92:0/433991608 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417e000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f41781f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 -- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4178067ad0 msgr2=0x7f4178141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4178067ad0 0x7f4178141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 -- 172.21.15.92:0/433991608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 msgr2=0x7f41781f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f41781f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.778+0000 7f417cc00640 1 -- 172.21.15.92:0/433991608 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780cfb50 con 0x7f4178154cd0 2024-03-19T02:55:02.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f417cc00640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41640090b0 tx=0x7f4164004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164047070 con 0x7f4178154cd0 2024-03-19T02:55:02.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4164037d50 con 0x7f4178154cd0 2024-03-19T02:55:02.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41780d0690 con 0x7f4178154cd0 2024-03-19T02:55:02.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164004270 con 0x7f4178154cd0 2024-03-19T02:55:02.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.779+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41780d0c50 con 0x7f4178154cd0 2024-03-19T02:55:02.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.780+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41780c6ee0 con 0x7f4178154cd0 2024-03-19T02:55:02.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.781+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f416404a070 con 0x7f4178154cd0 2024-03-19T02:55:02.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.782+0000 7f4175600640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 0x7f415806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:02.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.782+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f41640acd20 con 0x7f4178154cd0 2024-03-19T02:55:02.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.783+0000 7f417d600640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 0x7f415806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:02.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.783+0000 7f417d600640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 0x7f415806e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4170000f60 tx=0x7f4170009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:02.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.788+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f41780c6ee0 con 0x7f4178154cd0 2024-03-19T02:55:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.973+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41780ca8f0 con 0x7f4178154cd0 2024-03-19T02:55:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.975+0000 7f4175600640 1 -- 172.21.15.92:0/433991608 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f41780cfb50 con 0x7f4178154cd0 2024-03-19T02:55:02.976 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:02.976 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:02.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 msgr2=0x7f415806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 0x7f415806e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4170000f60 tx=0x7f4170009040 comp rx=0 tx=0).stop 2024-03-19T02:55:02.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 msgr2=0x7f41781f0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41640090b0 tx=0x7f4164004730 comp rx=0 tx=0).stop 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 shutdown_connections 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f415806bc00 0x7f415806e0c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41781550b0 0x7f41781f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4178154cd0 0x7f41781f0ba0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 --2- 172.21.15.92:0/433991608 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4178067ad0 0x7f4178141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 >> 172.21.15.92:0/433991608 conn(0x7f4178072970 msgr2=0x7f41781534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 shutdown_connections 2024-03-19T02:55:02.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:02.978+0000 7f4180000640 1 -- 172.21.15.92:0/433991608 wait complete. 2024-03-19T02:55:02.996 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-19T02:55:02.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:03.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.198+0000 7f5e19000640 1 Processor -- start 2024-03-19T02:55:03.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.199+0000 7f5e19000640 1 -- start start 2024-03-19T02:55:03.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e19000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5e14152bf0 0x7f5e1406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e19000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e140c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e19000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e140c9a80 con 0x7f5e140c6470 2024-03-19T02:55:03.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e19000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5e140c6c90 con 0x7f5e14152bf0 2024-03-19T02:55:03.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e19000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5e14159910 con 0x7f5e141593d0 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e11e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5e14152bf0 0x7f5e1406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e11400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5e140c6470 0x7f5e140c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46854/0 (socket says 172.21.15.92:46854) 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e11400640 1 -- 172.21.15.92:0/4068019778 learned_addr learned my addr 172.21.15.92:0/4068019778 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.200+0000 7f5e12800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e140c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e10a00640 1 -- 172.21.15.92:0/4068019778 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396061531 0 0) 0x7f5e140c9a80 con 0x7f5e140c6470 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e10a00640 1 -- 172.21.15.92:0/4068019778 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df8003520 con 0x7f5e140c6470 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e11e00640 1 -- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 msgr2=0x7f5e140c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e11e00640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e140c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e11e00640 1 -- 172.21.15.92:0/4068019778 >> v1:172.21.15.159:6789/0 conn(0x7f5e140c6470 legacy=0x7f5e140c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e11e00640 1 -- 172.21.15.92:0/4068019778 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e140c8650 con 0x7f5e14152bf0 2024-03-19T02:55:03.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.201+0000 7f5e12800640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e140c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.202+0000 7f5e11e00640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e14152bf0 0x7f5e1406eef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53d07cb76a17501d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.202+0000 7f5e10a00640 1 -- 172.21.15.92:0/4068019778 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04039070 con 0x7f5e14152bf0 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.202+0000 7f5e10a00640 1 -- 172.21.15.92:0/4068019778 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e0402cbd0 con 0x7f5e14152bf0 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.202+0000 7f5e10a00640 1 -- 172.21.15.92:0/4068019778 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04034440 con 0x7f5e14152bf0 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 -- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e14152bf0 msgr2=0x7f5e1406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e14152bf0 0x7f5e1406eef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402be20 comp rx=0 tx=0).stop 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 -- 172.21.15.92:0/4068019778 shutdown_connections 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e140c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 --2- 172.21.15.92:0/4068019778 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e14152bf0 0x7f5e1406eef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 -- 172.21.15.92:0/4068019778 >> 172.21.15.92:0/4068019778 conn(0x7f5e140c8d90 msgr2=0x7f5e140c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 -- 172.21.15.92:0/4068019778 shutdown_connections 2024-03-19T02:55:03.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.203+0000 7f5e19000640 1 -- 172.21.15.92:0/4068019778 wait complete. 2024-03-19T02:55:03.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.205+0000 7f5e19000640 1 Processor -- start 2024-03-19T02:55:03.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.205+0000 7f5e19000640 1 -- start start 2024-03-19T02:55:03.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e140c6470 0x7f5e141eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e1414b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1406ea60 con 0x7f5e14152bf0 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1406e4a0 con 0x7f5e140c6470 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e19000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1406e6f0 con 0x7f5e141593d0 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e11400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e11400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47854/0 (socket says 172.21.15.92:47854) 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e11400640 1 -- 172.21.15.92:0/2065390478 learned_addr learned my addr 172.21.15.92:0/2065390478 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e11e00640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e140c6470 0x7f5e141eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.206+0000 7f5e12800640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e1414b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 -- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 msgr2=0x7f5e1414b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e1414b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 -- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e140c6470 msgr2=0x7f5e141eeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e140c6470 0x7f5e141eeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 -- 172.21.15.92:0/2065390478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1414a030 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e11400640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e0000c460 tx=0x7f5e0000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00017070 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e00005ce0 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e000072f0 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1414a2d0 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.207+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e1414ae60 con 0x7f5e14152bf0 2024-03-19T02:55:03.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.209+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e140684c0 con 0x7f5e14152bf0 2024-03-19T02:55:03.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.209+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e0001a070 con 0x7f5e14152bf0 2024-03-19T02:55:03.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.210+0000 7f5e0a000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 0x7f5df406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.210+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5e00089e40 con 0x7f5e14152bf0 2024-03-19T02:55:03.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.211+0000 7f5e11e00640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 0x7f5df406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.211+0000 7f5e11e00640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 0x7f5df406e0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5e04036000 tx=0x7f5e04030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.215+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e00058250 con 0x7f5e14152bf0 2024-03-19T02:55:03.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.399+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e140c3b50 con 0x7f5e14152bf0 2024-03-19T02:55:03.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.402+0000 7f5e0a000640 1 -- 172.21.15.92:0/2065390478 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f5e0005c2f0 con 0x7f5e14152bf0 2024-03-19T02:55:03.404 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:03.404 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:03.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.404+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 msgr2=0x7f5df406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 0x7f5df406e0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5e04036000 tx=0x7f5e04030000 comp rx=0 tx=0).stop 2024-03-19T02:55:03.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 msgr2=0x7f5e140cb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e0000c460 tx=0x7f5e0000c860 comp rx=0 tx=0).stop 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 shutdown_connections 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5df406bc00 0x7f5df406e0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e141593d0 0x7f5e1414b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e14152bf0 0x7f5e140cb460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 --2- 172.21.15.92:0/2065390478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e140c6470 0x7f5e141eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 >> 172.21.15.92:0/2065390478 conn(0x7f5e140c8d90 msgr2=0x7f5e140c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 shutdown_connections 2024-03-19T02:55:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.405+0000 7f5e19000640 1 -- 172.21.15.92:0/2065390478 wait complete. 2024-03-19T02:55:03.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-19T02:55:03.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-19T02:55:03.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-19T02:55:03.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-19T02:55:03.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-19T02:55:03.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-19T02:55:03.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-19T02:55:03.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-19T02:55:03.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.711+0000 7f7e89000640 1 Processor -- start 2024-03-19T02:55:03.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.711+0000 7f7e89000640 1 -- start start 2024-03-19T02:55:03.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.713+0000 7f7e89000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7e7c038d60 0x7f7e7c0e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.713+0000 7f7e89000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7c0376e0 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.713+0000 7f7e89000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7c0502d0 con 0x7f7e7c037180 2024-03-19T02:55:03.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.713+0000 7f7e89000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7e7c0505b0 con 0x7f7e7c038d60 2024-03-19T02:55:03.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.715+0000 7f7e82800640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7e7c037180 0x7f7e7c0e2850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46862/0 (socket says 172.21.15.92:46862) 2024-03-19T02:55:03.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.715+0000 7f7e82800640 1 -- 172.21.15.92:0/1024104700 learned_addr learned my addr 172.21.15.92:0/1024104700 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.715+0000 7f7e81400640 1 --2- 172.21.15.92:0/1024104700 >> v2:172.21.15.92:3300/0 conn(0x7f7e7c038d60 0x7f7e7c0e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567165271 0 0) 0x7f7e7c0502d0 con 0x7f7e7c037180 2024-03-19T02:55:03.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e68003520 con 0x7f7e7c037180 2024-03-19T02:55:03.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030885371 0 0) 0x7f7e7c0376e0 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e7c0502d0 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046856285 0 0) 0x7f7e7c0502d0 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.716+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 >> v2:172.21.15.92:3300/0 conn(0x7f7e7c038d60 msgr2=0x7f7e7c0e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 --2- 172.21.15.92:0/1024104700 >> v2:172.21.15.92:3300/0 conn(0x7f7e7c038d60 0x7f7e7c0e2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 >> v1:172.21.15.159:6789/0 conn(0x7f7e7c037180 legacy=0x7f7e7c0e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c050890 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7f7e700016c0 con 0x7f7e7c0d92c0 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 >> v1:172.21.15.92:6789/0 conn(0x7f7e7c0d92c0 legacy=0x7f7e7c038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e68004ba0 0x7f7e68004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e680037a0 0x7f7e68003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c0505b0 con 0x7f7e68004ba0 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6800bf60 con 0x7f7e680037a0 2024-03-19T02:55:03.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6800c160 con 0x7f7e68004160 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.717+0000 7f7e82800640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e81e00640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e68004ba0 0x7f7e68004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e81400640 1 --2- 172.21.15.92:0/1024104700 >> v2:172.21.15.92:3300/0 conn(0x7f7e7c038d60 0x7f7e7c0e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e82800640 1 -- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e680037a0 msgr2=0x7f7e68003c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e82800640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e680037a0 0x7f7e68003c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e82800640 1 -- 172.21.15.92:0/1024104700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e68004ba0 msgr2=0x7f7e68004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.718+0000 7f7e82800640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e68004ba0 0x7f7e68004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.719+0000 7f7e82800640 1 -- 172.21.15.92:0/1024104700 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c050890 con 0x7f7e68004160 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.719+0000 7f7e81e00640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e68004ba0 0x7f7e68004fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.719+0000 7f7e82800640 1 -- 172.21.15.92:0/1024104700 reap_dead start 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.720+0000 7f7e82800640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e6800b9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e84077890 tx=0x7f7e68005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c729d79a7b6fbef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.720+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e840b4070 con 0x7f7e68004160 2024-03-19T02:55:03.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.720+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e840af620 con 0x7f7e68004160 2024-03-19T02:55:03.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.720+0000 7f7e80a00640 1 -- 172.21.15.92:0/1024104700 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70007ca0 con 0x7f7e68004160 2024-03-19T02:55:03.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.721+0000 7f7e89000640 1 -- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 msgr2=0x7f7e6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.721+0000 7f7e89000640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e6800b9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e84077890 tx=0x7f7e68005830 comp rx=0 tx=0).stop 2024-03-19T02:55:03.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.721+0000 7f7e89000640 1 -- 172.21.15.92:0/1024104700 shutdown_connections 2024-03-19T02:55:03.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.721+0000 7f7e89000640 1 --2- 172.21.15.92:0/1024104700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e6800b9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.721+0000 7f7e89000640 1 -- 172.21.15.92:0/1024104700 >> 172.21.15.92:0/1024104700 conn(0x7f7e7c054610 msgr2=0x7f7e7c054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.722+0000 7f7e89000640 1 -- 172.21.15.92:0/1024104700 shutdown_connections 2024-03-19T02:55:03.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.722+0000 7f7e89000640 1 -- 172.21.15.92:0/1024104700 wait complete. 2024-03-19T02:55:03.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.725+0000 7f7e89000640 1 Processor -- start 2024-03-19T02:55:03.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.725+0000 7f7e89000640 1 -- start start 2024-03-19T02:55:03.728 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.727+0000 7fbfb7400640 1 Processor -- start 2024-03-19T02:55:03.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.727+0000 7f7e89000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e7c04d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7f7e89000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e7c04cd40 0x7f7e7c174870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7f7e89000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 0x7f7e7c04dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7f7e82800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 0x7f7e7c04dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7f7e82800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 0x7f7e7c04dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60766/0 (socket says 172.21.15.92:60766) 2024-03-19T02:55:03.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7fcc2dc00640 1 Processor -- start 2024-03-19T02:55:03.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7f7e82800640 1 -- 172.21.15.92:0/3254963707 learned_addr learned my addr 172.21.15.92:0/3254963707 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7fcc2dc00640 1 -- start start 2024-03-19T02:55:03.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c037900 con 0x7f7e7c04cd40 2024-03-19T02:55:03.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c038130 con 0x7f7e7c174db0 2024-03-19T02:55:03.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c037a80 con 0x7f7e68004160 2024-03-19T02:55:03.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e82800640 1 -- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 msgr2=0x7f7e7c04d360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:55:03.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7fc2c0a00640 1 Processor -- start 2024-03-19T02:55:03.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7fc2c0a00640 1 -- start start 2024-03-19T02:55:03.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e82800640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e68004160 0x7f7e7c04d360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.728+0000 7fbfb7400640 1 -- start start 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfb7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8046490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfb7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 0x7fbfa804beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfb7400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa80dd7d0 con 0x7fbfa80e30c0 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfb7400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa8045480 con 0x7fbfa804ba90 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfb7400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa8046000 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.733 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.729+0000 7f7e82800640 1 -- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e7c04cd40 msgr2=0x7f7e7c174870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:55:03.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.730+0000 7fbfafe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 0x7fbfa804beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7fbfb5400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbfa80e30c0 0x7fbfa80e3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57912/0 (socket says 172.21.15.92:57912) 2024-03-19T02:55:03.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7fbfb4a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8046490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7fbfb5400640 1 -- 172.21.15.92:0/3709977948 learned_addr learned my addr 172.21.15.92:0/3709977948 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7f7e81400640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e7c04cd40 0x7f7e7c174870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7fcc2dc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcc28162440 0x7fcc2816a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fcc2dc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc281628b0 con 0x7fcc28158ed0 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fcc2dc00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc280c8a00 con 0x7fcc281592b0 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fcc2dc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcc280c67e0 con 0x7fcc28162440 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7f7e82800640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e7c04cd40 0x7f7e7c174870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7f7e82800640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c175390 con 0x7f7e7c174db0 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7f7e81400640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e7c04cd40 0x7f7e7c174870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fcc27400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fcc281592b0 0x7fcc2806cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46884/0 (socket says 172.21.15.92:46884) 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fcc27400640 1 -- 172.21.15.92:0/4275981817 learned_addr learned my addr 172.21.15.92:0/4275981817 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.731+0000 7fc2c0a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2bc0c8790 0x7fc2bc0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fc2c0a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2bc0c74e0 con 0x7fc2bc16b9b0 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fc2c0a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2bc16aed0 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fc2c0a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc2bc0c67e0 con 0x7fc2bc0c8790 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fc2b9e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc2bc0c42f0 0x7fc2bc167830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46874/0 (socket says 172.21.15.92:46874) 2024-03-19T02:55:03.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fc2b9e00640 1 -- 172.21.15.92:0/698927316 learned_addr learned my addr 172.21.15.92:0/698927316 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fbfaf400640 1 -- 172.21.15.92:0/3709977948 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085586311 0 0) 0x7fbfb0050470 con 0x7fbfa80e30c0 2024-03-19T02:55:03.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fbfb4a00640 1 -- 172.21.15.92:0/3709977948 >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 msgr2=0x7fbfa804beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7f9b62200640 1 Processor -- start 2024-03-19T02:55:03.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fcc2d200640 1 --2- 172.21.15.92:0/4275981817 >> v2:172.21.15.92:3300/0 conn(0x7fcc28162440 0x7fcc2816a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7f9b62200640 1 -- start start 2024-03-19T02:55:03.738 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071964085 0 0) 0x7fcc280c8a00 con 0x7fcc281592b0 2024-03-19T02:55:03.738 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0c003520 con 0x7fcc281592b0 2024-03-19T02:55:03.738 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509709424 0 0) 0x7fc2bc16aed0 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.738 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2a0003520 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.738 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.733+0000 7fc2ba800640 1 --2- 172.21.15.92:0/698927316 >> v2:172.21.15.92:3300/0 conn(0x7fc2bc0c8790 0x7fc2bc0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fbfb4a00640 1 --2- 172.21.15.92:0/3709977948 >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 0x7fbfa804beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fbfb4a00640 1 -- 172.21.15.92:0/3709977948 >> v1:172.21.15.92:6789/0 conn(0x7fbfa80e30c0 legacy=0x7fbfa80e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.732+0000 7fbfb4a00640 1 -- 172.21.15.92:0/3709977948 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80e3ec0 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7f7e82800640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 0x7f7e7c04dbd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e8404faa0 tx=0x7f7e84073fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3714101625 0 0) 0x7fc2a0003520 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.739 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 >> v2:172.21.15.92:3300/0 conn(0x7fc2bc0c8790 msgr2=0x7fc2bc0c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 --2- 172.21.15.92:0/698927316 >> v2:172.21.15.92:3300/0 conn(0x7fc2bc0c8790 0x7fc2bc0c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 >> v1:172.21.15.92:6789/0 conn(0x7fc2bc16b9b0 legacy=0x7fc2bc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc16b1b0 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7fc2a8001670 con 0x7fc2bc0c42f0 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 >> v1:172.21.15.159:6789/0 conn(0x7fc2bc0c42f0 legacy=0x7fc2bc167830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fb36f200640 1 Processor -- start 2024-03-19T02:55:03.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e840b4040 con 0x7f7e7c174db0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e840c2070 con 0x7f7e7c174db0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e840504d0 con 0x7f7e7c174db0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2955251677 0 0) 0x7fcc0c003520 con 0x7fcc281592b0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 >> v2:172.21.15.92:3300/0 conn(0x7fcc28162440 msgr2=0x7fcc2816a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 --2- 172.21.15.92:0/4275981817 >> v2:172.21.15.92:3300/0 conn(0x7fcc28162440 0x7fcc2816a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 >> v1:172.21.15.92:6789/0 conn(0x7fcc28158ed0 legacy=0x7fcc280c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc280c7160 con 0x7fcc281592b0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7fcc14001670 con 0x7fcc281592b0 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 >> v1:172.21.15.159:6789/0 conn(0x7fcc281592b0 legacy=0x7fcc2806cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 0x7fcc0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c004800 0x7fcc0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc0c005210 0x7fcc0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc280c67e0 con 0x7fcc0c0039b0 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c00c120 con 0x7fcc0c005210 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c00c320 con 0x7fcc0c004800 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fbfafe00640 1 --2- 172.21.15.92:0/3709977948 >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 0x7fbfa804beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a00039b0 0x7fc2a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 0x7fc2a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.742 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a00051c0 0x7fc2a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0c67e0 con 0x7fc2a00039b0 2024-03-19T02:55:03.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a000c0f0 con 0x7fc2a0004800 2024-03-19T02:55:03.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a000c2f0 con 0x7fc2a00051c0 2024-03-19T02:55:03.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 0x7fcc0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fbfb4a00640 1 --2- 172.21.15.92:0/3709977948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8046490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40097a0 tx=0x7fbfa402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed5febf6b3909bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc2d200640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc0c005210 0x7fcc0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27400640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c004800 0x7fcc0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 -- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c004800 msgr2=0x7fcc0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c004800 0x7fcc0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 -- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc0c005210 msgr2=0x7fcc0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc0c005210 0x7fcc0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.744 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fcc27e00640 1 -- 172.21.15.92:0/4275981817 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc280c7160 con 0x7fcc0c0039b0 2024-03-19T02:55:03.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.735+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 0x7fc2a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2bb200640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a00051c0 0x7fc2a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fcc2d200640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc0c005210 0x7fcc0c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.734+0000 7fb36f200640 1 -- start start 2024-03-19T02:55:03.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c175670 con 0x7f7e7c174db0 2024-03-19T02:55:03.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fcc2d200640 1 -- 172.21.15.92:0/4275981817 reap_dead start 2024-03-19T02:55:03.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fcc27400640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c004800 0x7fcc0c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fbfaf400640 1 -- 172.21.15.92:0/3709977948 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4035020 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.747 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fbfaf400640 1 -- 172.21.15.92:0/3709977948 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa4031830 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.747 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fbfaf400640 1 -- 172.21.15.92:0/3709977948 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4031c40 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.747 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c04c540 con 0x7f7e7c174db0 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7f9b62200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b540df690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b62200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b5404cc10 con 0x7f9b540dd620 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b62200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b5404e9a0 con 0x7f9b540d5e20 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b62200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9b540092d0 con 0x7f9b5404c660 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b60e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9b540d5e20 0x7f9b540dcf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46892/0 (socket says 172.21.15.92:46892) 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b60e00640 1 -- 172.21.15.92:0/4087874249 learned_addr learned my addr 172.21.15.92:0/4087874249 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113574500 0 0) 0x7f9b5404e9a0 con 0x7f9b540d5e20 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7fb36f200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3680dd770 0x7fb36804e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7fb36f200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 0x7fb3680e3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7fb36f200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36804e870 con 0x7fb368007e50 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7fb36f200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fb36804eb50 con 0x7fb3680dd770 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7fb36f200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb368008680 con 0x7fb36804cd20 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2b9e00640 1 -- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a00051c0 msgr2=0x7fc2a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a00051c0 0x7fc2a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2b9e00640 1 -- 172.21.15.92:0/698927316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a00039b0 msgr2=0x7fc2a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a00039b0 0x7fc2a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.736+0000 7fc2b9e00640 1 -- 172.21.15.92:0/698927316 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc16b1b0 con 0x7fc2a0004800 2024-03-19T02:55:03.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fcc27e00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 0x7fcc0c003dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcc18001730 tx=0x7fcc0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1046eb2afed74dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.750 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fc2bb200640 1 -- 172.21.15.92:0/698927316 reap_dead start 2024-03-19T02:55:03.750 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b44003520 con 0x7f9b540d5e20 2024-03-19T02:55:03.750 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.737+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e7c005390 con 0x7f7e7c174db0 2024-03-19T02:55:03.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f9b61800640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b540df690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f2d66000640 1 Processor -- start 2024-03-19T02:55:03.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fb36de00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3680dd770 0x7fb36804e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fb36de00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb3680dd770 0x7fb36804e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60802/0 (socket says 172.21.15.92:60802) 2024-03-19T02:55:03.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fb36de00640 1 -- 172.21.15.92:0/2548923722 learned_addr learned my addr 172.21.15.92:0/2548923722 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fbfb7400640 1 -- 172.21.15.92:0/3709977948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 msgr2=0x7fbfa8046490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fbfb7400640 1 --2- 172.21.15.92:0/3709977948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8046490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40097a0 tx=0x7fbfa402b8d0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3009083065 0 0) 0x7f9b44003520 con 0x7f9b540d5e20 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 msgr2=0x7f9b540df690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f9b5b400640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b540df690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 >> v1:172.21.15.92:6789/0 conn(0x7f9b540dd620 legacy=0x7f9b540def80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7f2d66000640 1 -- start start 2024-03-19T02:55:03.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fb36de00640 1 -- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 msgr2=0x7fb3680e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.738+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18001350 con 0x7fcc0c0039b0 2024-03-19T02:55:03.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fb36e800640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 0x7fb3680e3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fb36de00640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 0x7fb3680e3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 0x7fc2a0004c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c210 tx=0x7fc2a0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8befdb8230ab54a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fb36de00640 1 -- 172.21.15.92:0/2548923722 >> v1:172.21.15.92:6789/0 conn(0x7fb368007e50 legacy=0x7fb36804d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b54008700 con 0x7f9b540d5e20 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7f9b50001670 con 0x7f9b540d5e20 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 >> v1:172.21.15.159:6789/0 conn(0x7f9b540d5e20 legacy=0x7f9b540dcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b440039b0 0x7f9b44003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b44004800 0x7f9b44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b44005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b540092d0 con 0x7f9b440039b0 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4400baf0 con 0x7f9b44004800 2024-03-19T02:55:03.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4400bcf0 con 0x7f9b5404c660 2024-03-19T02:55:03.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8007d00 con 0x7fc2a0004800 2024-03-19T02:55:03.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b440039b0 0x7f9b44003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8030070 con 0x7fc2a0004800 2024-03-19T02:55:03.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.739+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc180052f0 con 0x7fcc0c0039b0 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fcc26a00640 1 -- 172.21.15.92:0/4275981817 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18003830 con 0x7fcc0c0039b0 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fbfb7400640 1 -- 172.21.15.92:0/3709977948 shutdown_connections 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fbfb7400640 1 --2- 172.21.15.92:0/3709977948 >> v2:172.21.15.92:3300/0 conn(0x7fbfa804ba90 0x7fbfa804beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fbfb7400640 1 --2- 172.21.15.92:0/3709977948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8046490 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fbfb7400640 1 -- 172.21.15.92:0/3709977948 >> 172.21.15.92:0/3709977948 conn(0x7fbfa80501b0 msgr2=0x7fbfa80505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fb36de00640 1 -- 172.21.15.92:0/2548923722 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb368008960 con 0x7fb3680dd770 2024-03-19T02:55:03.757 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7fb36e800640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 0x7fb3680e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f9b61800640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b44005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d66000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d600cc7e0 0x7f2d600cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.758 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d66000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cd3a0 0x7f2d6015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d66000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d601553f0 con 0x7f2d600ca2f0 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d66000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f2d600b39c0 con 0x7f2d600cc7e0 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d66000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6015ef00 con 0x7f2d600cd3a0 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f2d5fe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d600cc7e0 0x7f2d600cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2d600cc7e0 0x7f2d600cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60818/0 (socket says 172.21.15.92:60818) 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 -- 172.21.15.92:0/1098341004 learned_addr learned my addr 172.21.15.92:0/1098341004 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2b9400640 1 -- 172.21.15.92:0/698927316 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a802cdc0 con 0x7fc2a0004800 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 -- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 msgr2=0x7fc2a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 0x7fc2a0004c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c210 tx=0x7fc2a0005830 comp rx=0 tx=0).stop 2024-03-19T02:55:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d65600640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cd3a0 0x7f2d6015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 -- 172.21.15.92:0/698927316 shutdown_connections 2024-03-19T02:55:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 --2- 172.21.15.92:0/698927316 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2a0004800 0x7fc2a0004c80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 -- 172.21.15.92:0/698927316 >> 172.21.15.92:0/698927316 conn(0x7fc2bc06bac0 msgr2=0x7fc2bc0cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 -- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cd3a0 msgr2=0x7f2d6015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cd3a0 0x7f2d6015adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 -- 172.21.15.92:0/1098341004 >> v1:172.21.15.92:6789/0 conn(0x7f2d600ca2f0 legacy=0x7f2d600b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f2d5fe00640 1 -- 172.21.15.92:0/1098341004 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d600ca6d0 con 0x7f2d600cc7e0 2024-03-19T02:55:03.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 -- 172.21.15.92:0/698927316 shutdown_connections 2024-03-19T02:55:03.761 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.740+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7e840e3030 con 0x7f7e7c174db0 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7fc2c0a00640 1 -- 172.21.15.92:0/698927316 wait complete. 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f7e7a000640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e6406bc20 0x7f7e6406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7e84128ff0 con 0x7f7e7c174db0 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f7e7a000640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f7e64073ea0 0x7f7e640762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e840c8e80 con 0x7f7e64073ea0 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7e84129440 con 0x7f7e7c174db0 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f7e81e00640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e6406bc20 0x7f7e6406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fcc2dc00640 1 -- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 msgr2=0x7fcc0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fcc2dc00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 0x7fcc0c003dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcc18001730 tx=0x7fcc0c005830 comp rx=0 tx=0).stop 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fcc2dc00640 1 -- 172.21.15.92:0/4275981817 shutdown_connections 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fcc2dc00640 1 --2- 172.21.15.92:0/4275981817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0c0039b0 0x7fcc0c003dd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fcc2dc00640 1 -- 172.21.15.92:0/4275981817 >> 172.21.15.92:0/4275981817 conn(0x7fcc2806bac0 msgr2=0x7fcc280cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f9b60e00640 1 -- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 msgr2=0x7f9b44005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b44005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f9b60e00640 1 -- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b44004800 msgr2=0x7f9b44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b44004800 0x7f9b44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.741+0000 7f9b60e00640 1 -- 172.21.15.92:0/4087874249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b54008700 con 0x7f9b440039b0 2024-03-19T02:55:03.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f9b61800640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b5404c660 0x7f9b44005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fb36de00640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680dd770 0x7fb36804e1c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3600098a0 tx=0x7fb36002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6567fc37880f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fb36ca00640 1 -- 172.21.15.92:0/2548923722 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360035020 con 0x7fb3680dd770 2024-03-19T02:55:03.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fb36ca00640 1 -- 172.21.15.92:0/2548923722 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb360031640 con 0x7fb3680dd770 2024-03-19T02:55:03.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7fb36ca00640 1 -- 172.21.15.92:0/2548923722 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36002d040 con 0x7fb3680dd770 2024-03-19T02:55:03.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b440039b0 0x7f9b44003df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c260 tx=0x7f9b44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5c40cc67d5cc8b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fbfb7400640 1 -- 172.21.15.92:0/3709977948 shutdown_connections 2024-03-19T02:55:03.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fcc2dc00640 1 -- 172.21.15.92:0/4275981817 shutdown_connections 2024-03-19T02:55:03.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.742+0000 7f7e81400640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f7e64073ea0 0x7f7e640762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fbfb7400640 1 -- 172.21.15.92:0/3709977948 wait complete. 2024-03-19T02:55:03.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fcc2dc00640 1 -- 172.21.15.92:0/4275981817 wait complete. 2024-03-19T02:55:03.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f2d5fe00640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cc7e0 0x7f2d600cce60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d600c7490 tx=0x7f2d4c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d26325338cd0f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50007d10 con 0x7f9b440039b0 2024-03-19T02:55:03.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fb36f200640 1 -- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680dd770 msgr2=0x7fb36804e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7fb36f200640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680dd770 0x7fb36804e1c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3600098a0 tx=0x7fb36002b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b50030070 con 0x7f9b440039b0 2024-03-19T02:55:03.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f7e81e00640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e6406bc20 0x7f7e6406e0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001eb0 tx=0x7f7e70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.743+0000 7f9b5b400640 1 -- 172.21.15.92:0/4087874249 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5002ce20 con 0x7f9b440039b0 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7fb36f200640 1 -- 172.21.15.92:0/2548923722 shutdown_connections 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7fb36f200640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36804cd20 0x7fb3680e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7fb36f200640 1 --2- 172.21.15.92:0/2548923722 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3680dd770 0x7fb36804e1c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7fb36f200640 1 -- 172.21.15.92:0/2548923722 >> 172.21.15.92:0/2548923722 conn(0x7fb36800e6a0 msgr2=0x7fb368009d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7f2d5f400640 1 -- 172.21.15.92:0/1098341004 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0317c0 con 0x7f2d600cc7e0 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7f2d5f400640 1 -- 172.21.15.92:0/1098341004 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4c037070 con 0x7f2d600cc7e0 2024-03-19T02:55:03.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7f2d5f400640 1 -- 172.21.15.92:0/1098341004 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c031b80 con 0x7f2d600cc7e0 2024-03-19T02:55:03.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7f7e81400640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f7e64073ea0 0x7f7e640762c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.744+0000 7fb36f200640 1 -- 172.21.15.92:0/2548923722 shutdown_connections 2024-03-19T02:55:03.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7fb36f200640 1 -- 172.21.15.92:0/2548923722 wait complete. 2024-03-19T02:55:03.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7fcc2dc00640 1 Processor -- start 2024-03-19T02:55:03.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7fcc2dc00640 1 -- start start 2024-03-19T02:55:03.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7fbfb7400640 1 Processor -- start 2024-03-19T02:55:03.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7f2d66000640 1 -- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cc7e0 msgr2=0x7f2d600cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.745+0000 7f2d66000640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cc7e0 0x7f2d600cce60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d600c7490 tx=0x7f2d4c02b860 comp rx=0 tx=0).stop 2024-03-19T02:55:03.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f9b62200640 1 -- 172.21.15.92:0/4087874249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b440039b0 msgr2=0x7f9b44003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7fbfb7400640 1 -- start start 2024-03-19T02:55:03.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f9b62200640 1 --2- 172.21.15.92:0/4087874249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b440039b0 0x7f9b44003df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c260 tx=0x7f9b44005830 comp rx=0 tx=0).stop 2024-03-19T02:55:03.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 -- 172.21.15.92:0/1098341004 shutdown_connections 2024-03-19T02:55:03.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cd3a0 0x7f2d6015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 --2- 172.21.15.92:0/1098341004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cc7e0 0x7f2d600cce60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 -- 172.21.15.92:0/1098341004 >> 172.21.15.92:0/1098341004 conn(0x7f2d60066f50 msgr2=0x7f2d60067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f9b61800640 1 -- 172.21.15.92:0/4087874249 reap_dead start 2024-03-19T02:55:03.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 -- 172.21.15.92:0/1098341004 shutdown_connections 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7fd288400640 1 Processor -- start 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7fd288400640 1 -- start start 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== osd.0 v2:172.21.15.92:6814/3305119790 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e840c8e80 con 0x7f7e64073ea0 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f9b62200640 1 -- 172.21.15.92:0/4087874249 shutdown_connections 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f9b62200640 1 -- 172.21.15.92:0/4087874249 >> 172.21.15.92:0/4087874249 conn(0x7f9b5404dd30 msgr2=0x7f9b5404e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7f9b62200640 1 -- 172.21.15.92:0/4087874249 shutdown_connections 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c0039b0 0x7fcc28161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2814fbd0 0x7fcc28156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc28156140 0x7fcc28157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2815efa0 con 0x7fcc2814fbd0 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcc280c7700 con 0x7fcc28156140 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2dc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2815f280 con 0x7fcc0c0039b0 2024-03-19T02:55:03.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7f9b62200640 1 -- 172.21.15.92:0/4087874249 wait complete. 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2d200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc28156140 0x7fcc28157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2d200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc28156140 0x7fcc28157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60834/0 (socket says 172.21.15.92:60834) 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc2d200640 1 -- 172.21.15.92:0/574607876 learned_addr learned my addr 172.21.15.92:0/574607876 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fcc27e00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c0039b0 0x7fcc28161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2801593f0 0x7fd2800cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 0x7fd28006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 0x7fd2800cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd28015fdd0 con 0x7fd28006cbc0 2024-03-19T02:55:03.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd2800cc410 con 0x7fd2801593f0 2024-03-19T02:55:03.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.747+0000 7fd288400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd280162510 con 0x7fd2800cbaa0 2024-03-19T02:55:03.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fb36f200640 1 Processor -- start 2024-03-19T02:55:03.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fb36f200640 1 -- start start 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2801593f0 0x7fd2800cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 0x7fd28006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd286400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 0x7fd2800cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2801593f0 0x7fd2800cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60838/0 (socket says 172.21.15.92:60838) 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd286400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 0x7fd2800cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37720/0 (socket says 172.21.15.92:37720) 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 -- 172.21.15.92:0/2844265870 learned_addr learned my addr 172.21.15.92:0/2844265870 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fcc27e00640 1 -- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc28156140 msgr2=0x7fcc28157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fc2c0a00640 1 Processor -- start 2024-03-19T02:55:03.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fcc27e00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc28156140 0x7fcc28157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 -- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 msgr2=0x7fd2800cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 --2- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 0x7fd2800cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fd285a00640 1 -- 172.21.15.92:0/2844265870 >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 msgr2=0x7fd28006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:03.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7fcc27e00640 1 -- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2814fbd0 msgr2=0x7fcc28156c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:55:03.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fd285a00640 1 --2- 172.21.15.92:0/2844265870 >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 0x7fd28006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fd285a00640 1 -- 172.21.15.92:0/2844265870 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2801627f0 con 0x7fd2801593f0 2024-03-19T02:55:03.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.746+0000 7f2d66000640 1 -- 172.21.15.92:0/1098341004 wait complete. 2024-03-19T02:55:03.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.748+0000 7f2d66000640 1 Processor -- start 2024-03-19T02:55:03.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fd285000640 1 --2- 172.21.15.92:0/2844265870 >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 0x7fd28006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fcc27400640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2814fbd0 0x7fcc28156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fcc27e00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2814fbd0 0x7fcc28156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7f9b62200640 1 Processor -- start 2024-03-19T02:55:03.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fcc27e00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc280c33c0 con 0x7fcc0c0039b0 2024-03-19T02:55:03.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fbfb7400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa804ba90 0x7fbfa8049d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fcc27400640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2814fbd0 0x7fcc28156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7f9b62200640 1 -- start start 2024-03-19T02:55:03.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fbfb7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8047ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fb36f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb368007e50 0x7fb3681799e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fbfb7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 0x7fbfa80481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fcc27e00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c0039b0 0x7fcc28161660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0057e0 tx=0x7fcc18001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fb36f200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36804cd20 0x7fb368035cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fb36f200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 0x7fb3680ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18041070 con 0x7fcc0c0039b0 2024-03-19T02:55:03.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fd285a00640 1 --2- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd2800cca50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd28006ca50 tx=0x7fd26c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b107c340a52b34ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb36804df10 con 0x7fb368007e50 2024-03-19T02:55:03.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36f200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb36804dd90 con 0x7fb36804cd20 2024-03-19T02:55:03.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36f200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb3680e3a80 con 0x7fb3680e0750 2024-03-19T02:55:03.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 0x7fb3680ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc180054a0 con 0x7fcc0c0039b0 2024-03-19T02:55:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18006c00 con 0x7fcc0c0039b0 2024-03-19T02:55:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 0x7fb3680ddbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37730/0 (socket says 172.21.15.92:37730) 2024-03-19T02:55:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 -- 172.21.15.92:0/3556920305 learned_addr learned my addr 172.21.15.92:0/3556920305 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fbfb4a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa804ba90 0x7fbfa8049d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 0x7fbfa80481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc281f8300 con 0x7fcc0c0039b0 2024-03-19T02:55:03.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fd27fe00640 1 -- 172.21.15.92:0/2844265870 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c034070 con 0x7fd2801593f0 2024-03-19T02:55:03.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc281f88c0 con 0x7fcc0c0039b0 2024-03-19T02:55:03.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fd27fe00640 1 -- 172.21.15.92:0/2844265870 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26c030860 con 0x7fd2801593f0 2024-03-19T02:55:03.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.749+0000 7fc2c0a00640 1 -- start start 2024-03-19T02:55:03.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fd27fe00640 1 -- 172.21.15.92:0/2844265870 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c030c30 con 0x7fd2801593f0 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a0004800 0x7fc2bc1762a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 0x7fc2bc160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2bc2063f0 0x7fc2bc0c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0c6e20 con 0x7fc2bc2063f0 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0c77c0 con 0x7fc2bc209af0 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7fc2c0a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc2bc0c6fa0 con 0x7fc2a0004800 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b440039b0 0x7f9b540d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 0x7f9b540498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b540d9840 0x7f9b540d9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b54008f60 con 0x7f9b540c4060 2024-03-19T02:55:03.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.750+0000 7f2d66000640 1 -- start start 2024-03-19T02:55:03.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9b54008c80 con 0x7f9b540d9840 2024-03-19T02:55:03.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7f9b62200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9b5404d830 con 0x7f9b440039b0 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fc2b9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 0x7fc2bc160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fc2b9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 0x7fc2bc160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60882/0 (socket says 172.21.15.92:60882) 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fc2b9e00640 1 -- 172.21.15.92:0/2665105773 learned_addr learned my addr 172.21.15.92:0/2665105773 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 0x7fbfa80481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47964/0 (socket says 172.21.15.92:47964) 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb5400640 1 -- 172.21.15.92:0/1005435231 learned_addr learned my addr 172.21.15.92:0/1005435231 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80459b0 con 0x7fbfa80e30c0 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80e3c60 con 0x7fbfa804ba90 2024-03-19T02:55:03.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa8045c90 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc2804f580 con 0x7fcc0c0039b0 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fc2bb200640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2bc2063f0 0x7fc2bc0c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 -- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a0004800 msgr2=0x7fc2bc1762a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2a0004800 0x7fc2bc1762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 -- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2bc2063f0 msgr2=0x7fc2bc0c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2bc2063f0 0x7fc2bc0c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc15f630 con 0x7fc2bc209af0 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2bb200640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2bc2063f0 0x7fc2bc0c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:03.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fc2b9e00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 0x7fc2bc160af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c330 tx=0x7fc2a8001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7f9b5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 0x7f9b540498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7f9b5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 0x7f9b540498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47978/0 (socket says 172.21.15.92:47978) 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7f9b5be00640 1 -- 172.21.15.92:0/4168610696 learned_addr learned my addr 172.21.15.92:0/4168610696 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b440039b0 0x7f9b540d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f9b5be00640 1 -- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b440039b0 msgr2=0x7f9b540d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f9b5be00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b440039b0 0x7f9b540d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f9b5be00640 1 -- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b540d9840 msgr2=0x7f9b540d9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f9b61800640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b540d9840 0x7f9b540d9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 -- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36804cd20 msgr2=0x7fb368035cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36804cd20 0x7fb368035cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 -- 172.21.15.92:0/3556920305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb368007e50 msgr2=0x7fb3681799e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb368007e50 0x7fb3681799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fb36e800640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36817d160 con 0x7fb3680e0750 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fb36e800640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 0x7fb3680ddbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb37006a040 tx=0x7fb370076c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 -- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 msgr2=0x7fd2800cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 --2- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd2800cca50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd28006ca50 tx=0x7fd26c02b9a0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 -- 172.21.15.92:0/2844265870 shutdown_connections 2024-03-19T02:55:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 --2- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2800cbaa0 0x7fd2800cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 --2- 172.21.15.92:0/2844265870 >> v2:172.21.15.92:3300/0 conn(0x7fd28006cbc0 0x7fd28006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 --2- 172.21.15.92:0/2844265870 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd2800cca50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.752+0000 7fd288400640 1 -- 172.21.15.92:0/2844265870 >> 172.21.15.92:0/2844265870 conn(0x7fd28006da80 msgr2=0x7fd28006c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fd288400640 1 -- 172.21.15.92:0/2844265870 shutdown_connections 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fd288400640 1 -- 172.21.15.92:0/2844265870 wait complete. 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.751+0000 7fbfafe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8047ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f2d66000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d600ca2f0 0x7f2d600cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7f2d66000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cc7e0 0x7f2d600cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d66000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 0x7f2d60200b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcc1803b040 con 0x7fcc0c0039b0 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80049f0 con 0x7fc2bc209af0 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8004b90 con 0x7fc2bc209af0 2024-03-19T02:55:03.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc160210 con 0x7fc2bc209af0 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfafe00640 1 -- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa804ba90 msgr2=0x7fbfa8049d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfafe00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa804ba90 0x7fbfa8049d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfafe00640 1 -- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 msgr2=0x7fbfa80481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfafe00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 0x7fbfa80481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfafe00640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80b7080 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfb5400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfa80e30c0 0x7fbfa80481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fbfb4a00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfa804ba90 0x7fbfa8049d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d66000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6015e8b0 con 0x7f2d600ca2f0 2024-03-19T02:55:03.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d66000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6015e5d0 con 0x7f2d600cd3a0 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d66000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6015eb90 con 0x7f2d600cc7e0 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cc7e0 0x7f2d600cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d65600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 0x7f2d60200b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d65600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 0x7f2d60200b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60898/0 (socket says 172.21.15.92:60898) 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f2d65600640 1 -- 172.21.15.92:0/1802354240 learned_addr learned my addr 172.21.15.92:0/1802354240 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370073560 con 0x7fb3680e0750 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb370073720 con 0x7fb3680e0750 2024-03-19T02:55:03.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.753+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37006d6a0 con 0x7fb3680e0750 2024-03-19T02:55:03.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8007070 con 0x7fc2bc209af0 2024-03-19T02:55:03.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc15ff30 con 0x7fc2bc209af0 2024-03-19T02:55:03.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fbfafe00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8047ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00e8a0 tx=0x7fbf9c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.754+0000 7f9b5be00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b540d9840 0x7f9b540d9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c017070 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c00c940 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00cd30 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7f2d65600640 1 -- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cc7e0 msgr2=0x7f2d600cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7f2d65600640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cc7e0 0x7f2d600cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb368034df0 con 0x7fb3680e0750 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb368178f00 con 0x7fb3680e0750 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fcc1fe00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc0806bca0 0x7fcc0806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcc180abf20 con 0x7fcc0c0039b0 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fcc1fe00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7fcc08073ec0 0x7fcc080762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc18009eb0 con 0x7fcc08073ec0 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fcc180ae580 con 0x7fcc0c0039b0 2024-03-19T02:55:03.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc294001030 con 0x7fc2bc209af0 2024-03-19T02:55:03.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa80b7360 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7f2d65600640 1 -- 172.21.15.92:0/1802354240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d600ca2f0 msgr2=0x7f2d600cf090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:55:03.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f2d64c00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d600ca2f0 0x7f2d600cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fd288400640 1 Processor -- start 2024-03-19T02:55:03.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.756+0000 7fd288400640 1 -- start start 2024-03-19T02:55:03.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa80b7920 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f2d65600640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d600ca2f0 0x7f2d600cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f2d65600640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d600d0720 con 0x7f2d600cd3a0 2024-03-19T02:55:03.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fcc27400640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc0806bca0 0x7fcc0806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f2d5fe00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d600cc7e0 0x7f2d600cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:03.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.755+0000 7f9b5be00640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b540d8e00 con 0x7f9b540c4060 2024-03-19T02:55:03.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f2d64c00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d600ca2f0 0x7f2d600cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7f9b61800640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b540d9840 0x7f9b540d9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fcc2d200640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7fcc08073ec0 0x7fcc080762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fd288400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd28006cbc0 0x7fd28015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fd288400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2800cbaa0 0x7fd2801567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fd288400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd280156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd288400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd280163090 con 0x7fd2800cbaa0 2024-03-19T02:55:03.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd288400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd280162db0 con 0x7fd2801593f0 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd288400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd280163370 con 0x7fd28006cbc0 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.757+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfa80053c0 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd285a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd28006cbc0 0x7fd28015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd286400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd280156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd285a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd28006cbc0 0x7fd28015f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37772/0 (socket says 172.21.15.92:37772) 2024-03-19T02:55:03.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd285a00640 1 -- 172.21.15.92:0/1665602631 learned_addr learned my addr 172.21.15.92:0/1665602631 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:03.806 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fd285000640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2800cbaa0 0x7fd2801567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.806 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7f9b5be00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 0x7f9b540498b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c530 tx=0x7f9b4800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.806 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.758+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3680057e0 con 0x7fb3680e0750 2024-03-19T02:55:03.806 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7f2d65600640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 0x7f2d60200b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e7f0 tx=0x7f2d5000ebf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b480052d0 con 0x7f9b540c4060 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 -- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd28006cbc0 msgr2=0x7fd28015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fcc2d200640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7fcc08073ec0 0x7fcc080762e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd28006cbc0 0x7fd28015f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 -- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 msgr2=0x7fd280156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd280156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2801ebef0 con 0x7fd2800cbaa0 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd286400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2801593f0 0x7fd280156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc2a8046030 con 0x7fc2bc209af0 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b48005470 con 0x7f9b540c4060 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b480052d0 con 0x7f9b540c4060 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbf9c01a070 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b540d90e0 con 0x7f9b540c4060 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50017070 con 0x7f2d600cd3a0 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5000ee40 con 0x7f2d600cd3a0 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000cb80 con 0x7f2d600cd3a0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb37006d840 con 0x7fb3680e0750 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.759+0000 7fd285000640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2800cbaa0 0x7fd2801567b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd27400e900 tx=0x7fd27400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274017070 con 0x7fd2800cbaa0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2801ec1d0 con 0x7fd2800cbaa0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2800d11d0 con 0x7fd2800cbaa0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27400ca00 con 0x7fd2800cbaa0 2024-03-19T02:55:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274021060 con 0x7fd2800cbaa0 2024-03-19T02:55:03.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b540d71b0 con 0x7f9b540c4060 2024-03-19T02:55:03.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fbfacc00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbf9006bcd0 0x7fbf9006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbf9c08ad60 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fbfb4a00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbf9006bcd0 0x7fbf9006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d600d09c0 con 0x7f2d600cd3a0 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.760+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d600ce5b0 con 0x7f2d600cd3a0 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fbfacc00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7fbf90073f50 0x7fbf90076370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf9c00e020 con 0x7fbf90073f50 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbf9c08b150 con 0x7fbfa80dd3f0 2024-03-19T02:55:03.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fb366000640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb35006bca0 0x7fb35006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== osd.1 v2:172.21.15.92:6808/3121211588 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc18009eb0 con 0x7fcc08073ec0 2024-03-19T02:55:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b540057e0 con 0x7f9b540c4060 2024-03-19T02:55:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d6004f580 con 0x7f2d600cd3a0 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.761+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb370106c90 con 0x7fb3680e0750 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fb366000640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7fb350073ec0 0x7fb3500762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb370062ac0 con 0x7fb350073ec0 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb3701092f0 con 0x7fb3680e0750 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.762+0000 7fb36de00640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb35006bca0 0x7fb35006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fb36de00640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb35006bca0 0x7fb35006e160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb360036000 tx=0x7fb360030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fb36d400640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7fb350073ec0 0x7fb3500762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fc2b2a00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc29c06bca0 0x7fc29c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc2a80abd80 con 0x7fc2bc209af0 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fc2b2a00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7fc29c073ec0 0x7fc29c0762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc29c076990 con 0x7fc29c073ec0 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.763+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc2a80ae360 con 0x7fc2bc209af0 2024-03-19T02:55:03.815 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.764+0000 7fb36d400640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7fb350073ec0 0x7fb3500762e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.764+0000 7fc2bb200640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7fc29c073ec0 0x7fc29c0762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.765+0000 7fc2bb200640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7fc29c073ec0 0x7fc29c0762e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.764+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b48012630 con 0x7f9b540c4060 2024-03-19T02:55:03.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.765+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== osd.6 v2:172.21.15.159:6800/2808929873 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb370062ac0 con 0x7fb350073ec0 2024-03-19T02:55:03.816 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7f9b58c00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b3c06bdd0 0x7f9b3c06e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.765+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd27401a070 con 0x7fd2800cbaa0 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd27d600640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd26006bc00 0x7fd26006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd25c001030 con 0x7fd2800cbaa0 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd27408a910 con 0x7fd2800cbaa0 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd285a00640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd26006bc00 0x7fd26006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd27d600640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7fd260073e00 0x7fd260076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2600768d0 con 0x7fd260073e00 2024-03-19T02:55:03.817 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd274054f20 con 0x7fd2800cbaa0 2024-03-19T02:55:03.818 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9b4808a2d0 con 0x7f9b540c4060 2024-03-19T02:55:03.818 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.766+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b3c06bdd0 0x7f9b3c06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.818 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.765+0000 7fbfb5400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7fbf90073f50 0x7fbf90076370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.818 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fbfb4a00640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbf9006bcd0 0x7fbf9006e190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4036000 tx=0x7fbfa4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7f9b58c00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7f9b3c073fd0 0x7f9b3c0763f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fc2ba800640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc29c06bca0 0x7fc29c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== osd.2 v2:172.21.15.92:6800/652641961 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc29c076990 con 0x7fc29c073ec0 2024-03-19T02:55:03.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fd286400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7fd260073e00 0x7fd260076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fd285a00640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd26006bc00 0x7fd26006e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009630 tx=0x7fd26c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b4800ee50 con 0x7f9b3c073fd0 2024-03-19T02:55:03.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9b48091050 con 0x7f9b540c4060 2024-03-19T02:55:03.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7fcc27400640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc0806bca0 0x7fcc0806e160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc14001eb0 tx=0x7fcc14009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7f9b61800640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7f9b3c073fd0 0x7f9b3c0763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.767+0000 7f9b60e00640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b3c06bdd0 0x7f9b3c06e290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b50034000 tx=0x7f9b5002f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.768+0000 7fbfb5400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7fbf90073f50 0x7fbf90076370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.768+0000 7fc2ba800640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc29c06bca0 0x7fc29c06e160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0c9e50 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.821 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.768+0000 7fd286400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7fd260073e00 0x7fd260076220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.768+0000 7f9b61800640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7f9b3c073fd0 0x7f9b3c0763f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.769+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== osd.4 v2:172.21.15.159:6808/4013217604 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b4800ee50 con 0x7f9b3c073fd0 2024-03-19T02:55:03.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.769+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== osd.3 v2:172.21.15.92:6818/1598837651 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf9c00e020 con 0x7fbf90073f50 2024-03-19T02:55:03.822 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.771+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d5001a070 con 0x7f2d600cd3a0 2024-03-19T02:55:03.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.771+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== osd.7 v2:172.21.15.159:6816/3107756136 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2600768d0 con 0x7fd260073e00 2024-03-19T02:55:03.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.772+0000 7f2d5cc00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2d4006bcd0 0x7f2d4006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.772+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2d5008a9b0 con 0x7f2d600cd3a0 2024-03-19T02:55:03.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.772+0000 7f2d5cc00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f2d40073f50 0x7f2d40076370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.773+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d40076a20 con 0x7f2d40073f50 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.773+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2d5008f2a0 con 0x7f2d600cd3a0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.773+0000 7f2d64c00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2d4006bcd0 0x7f2d4006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.773+0000 7f2d5fe00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f2d40073f50 0x7f2d40076370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.774+0000 7f2d64c00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2d4006bcd0 0x7f2d4006e190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d58001dc0 tx=0x7f2d58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.779+0000 7f2d5fe00640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f2d40073f50 0x7f2d40076370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.780+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== osd.5 v2:172.21.15.159:6824/1484023345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d40076a20 con 0x7f2d40073f50 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.781+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 --> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7e7c051fc0 con 0x7f7e64073ea0 2024-03-19T02:55:03.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.782+0000 7f7e7a000640 1 -- 172.21.15.92:0/3254963707 <== osd.0 v2:172.21.15.92:6814/3305119790 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7e7c051fc0 con 0x7f7e64073ea0 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.783+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f7e64073ea0 msgr2=0x7f7e640762c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.783+0000 7f7e89000640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f7e64073ea0 0x7f7e640762c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.783+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e6406bc20 msgr2=0x7f7e6406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7e6406bc20 0x7f7e6406e0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001eb0 tx=0x7f7e70009040 comp rx=0 tx=0).stop 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 msgr2=0x7f7e7c04dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 --2- 172.21.15.92:0/3254963707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e7c174db0 0x7f7e7c04dbd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e8404faa0 tx=0x7f7e84073fc0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e82800640 1 -- 172.21.15.92:0/3254963707 reap_dead start 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 shutdown_connections 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 >> 172.21.15.92:0/3254963707 conn(0x7f7e7c054610 msgr2=0x7f7e7c038650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.826 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.784+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 shutdown_connections 2024-03-19T02:55:03.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.785+0000 7f7e89000640 1 -- 172.21.15.92:0/3254963707 wait complete. 2024-03-19T02:55:03.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.795+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 --> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b540bece0 con 0x7f9b3c073fd0 2024-03-19T02:55:03.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.795+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 --> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb3681797a0 con 0x7fb350073ec0 2024-03-19T02:55:03.827 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.795+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 --> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc28155710 con 0x7fcc08073ec0 2024-03-19T02:55:03.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.796+0000 7f9b58c00640 1 -- 172.21.15.92:0/4168610696 <== osd.4 v2:172.21.15.159:6808/4013217604 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9b540bece0 con 0x7f9b3c073fd0 2024-03-19T02:55:03.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.796+0000 7fb366000640 1 -- 172.21.15.92:0/3556920305 <== osd.6 v2:172.21.15.159:6800/2808929873 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb3681797a0 con 0x7fb350073ec0 2024-03-19T02:55:03.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7f9b3c073fd0 msgr2=0x7f9b3c0763f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7f9b3c073fd0 0x7f9b3c0763f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7fb350073ec0 msgr2=0x7fb3500762e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7fb350073ec0 0x7fb3500762e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fcc1fe00640 1 -- 172.21.15.92:0/574607876 <== osd.1 v2:172.21.15.92:6808/3121211588 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcc28155710 con 0x7fcc08073ec0 2024-03-19T02:55:03.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b3c06bdd0 msgr2=0x7f9b3c06e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b3c06bdd0 0x7f9b3c06e290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b50034000 tx=0x7f9b5002f000 comp rx=0 tx=0).stop 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 msgr2=0x7f9b540498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b62200640 1 --2- 172.21.15.92:0/4168610696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b540c4060 0x7f9b540498b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c530 tx=0x7f9b4800c9f0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb35006bca0 msgr2=0x7fb35006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb35006bca0 0x7fb35006e160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb360036000 tx=0x7fb360030000 comp rx=0 tx=0).stop 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 msgr2=0x7fb3680ddbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36f200640 1 --2- 172.21.15.92:0/3556920305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3680e0750 0x7fb3680ddbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb37006a040 tx=0x7fb370076c70 comp rx=0 tx=0).stop 2024-03-19T02:55:03.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7fb36e800640 1 -- 172.21.15.92:0/3556920305 reap_dead start 2024-03-19T02:55:03.831 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.797+0000 7f9b61800640 1 -- 172.21.15.92:0/4168610696 reap_dead start 2024-03-19T02:55:03.831 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 shutdown_connections 2024-03-19T02:55:03.831 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 >> 172.21.15.92:0/4168610696 conn(0x7f9b5404dd30 msgr2=0x7f9b540d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.831 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7fcc08073ec0 msgr2=0x7fcc080762e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.831 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7fcc2dc00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7fcc08073ec0 0x7fcc080762e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.832 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 shutdown_connections 2024-03-19T02:55:03.832 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7f9b62200640 1 -- 172.21.15.92:0/4168610696 wait complete. 2024-03-19T02:55:03.832 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.798+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 shutdown_connections 2024-03-19T02:55:03.832 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 >> 172.21.15.92:0/3556920305 conn(0x7fb36800e6a0 msgr2=0x7fb36816c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc0806bca0 msgr2=0x7fcc0806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcc0806bca0 0x7fcc0806e160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc14001eb0 tx=0x7fcc14009040 comp rx=0 tx=0).stop 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c0039b0 msgr2=0x7fcc28161660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 --2- 172.21.15.92:0/574607876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc0c0039b0 0x7fcc28161660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0057e0 tx=0x7fcc18001120 comp rx=0 tx=0).stop 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2d200640 1 -- 172.21.15.92:0/574607876 reap_dead start 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 shutdown_connections 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 >> 172.21.15.92:0/574607876 conn(0x7fcc2806bac0 msgr2=0x7fcc28151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 shutdown_connections 2024-03-19T02:55:03.833 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.799+0000 7fb36f200640 1 -- 172.21.15.92:0/3556920305 wait complete. 2024-03-19T02:55:03.834 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.800+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 shutdown_connections 2024-03-19T02:55:03.834 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.800+0000 7fcc2dc00640 1 -- 172.21.15.92:0/574607876 wait complete. 2024-03-19T02:55:03.834 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.800+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 --> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc294002dd0 con 0x7fc29c073ec0 2024-03-19T02:55:03.834 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.800+0000 7fc2b2a00640 1 -- 172.21.15.92:0/2665105773 <== osd.2 v2:172.21.15.92:6800/652641961 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc294002dd0 con 0x7fc29c073ec0 2024-03-19T02:55:03.835 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.801+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 --> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfa80477f0 con 0x7fbf90073f50 2024-03-19T02:55:03.835 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfacc00640 1 -- 172.21.15.92:0/1005435231 <== osd.3 v2:172.21.15.92:6818/1598837651 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbfa80477f0 con 0x7fbf90073f50 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7fc29c073ec0 msgr2=0x7fc29c0762e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fc2c0a00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7fc29c073ec0 0x7fc29c0762e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7fbf90073f50 msgr2=0x7fbf90076370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7fbf90073f50 0x7fbf90076370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbf9006bcd0 msgr2=0x7fbf9006e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.836 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbf9006bcd0 0x7fbf9006e190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4036000 tx=0x7fbfa4008030 comp rx=0 tx=0).stop 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 msgr2=0x7fbfa8047ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.802+0000 7fbfb7400640 1 --2- 172.21.15.92:0/1005435231 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfa80dd3f0 0x7fbfa8047ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00e8a0 tx=0x7fbf9c00ed60 comp rx=0 tx=0).stop 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.803+0000 7fbfb5400640 1 -- 172.21.15.92:0/1005435231 reap_dead start 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.803+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 shutdown_connections 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.803+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 >> 172.21.15.92:0/1005435231 conn(0x7fbfa80501b0 msgr2=0x7fbfa8006a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 shutdown_connections 2024-03-19T02:55:03.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fbfb7400640 1 -- 172.21.15.92:0/1005435231 wait complete. 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.803+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc29c06bca0 msgr2=0x7fc29c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.803+0000 7fc2c0a00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc29c06bca0 0x7fc29c06e160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0c9e50 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 msgr2=0x7fc2bc160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fc2c0a00640 1 --2- 172.21.15.92:0/2665105773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2bc209af0 0x7fc2bc160af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c330 tx=0x7fc2a8001970 comp rx=0 tx=0).stop 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fc2bb200640 1 -- 172.21.15.92:0/2665105773 reap_dead start 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 shutdown_connections 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 >> 172.21.15.92:0/2665105773 conn(0x7fc2bc06bac0 msgr2=0x7fc2bc0ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.804+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 --> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd25c002dd0 con 0x7fd260073e00 2024-03-19T02:55:03.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.805+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 shutdown_connections 2024-03-19T02:55:03.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.805+0000 7fc2c0a00640 1 -- 172.21.15.92:0/2665105773 wait complete. 2024-03-19T02:55:03.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.806+0000 7fd27d600640 1 -- 172.21.15.92:0/1665602631 <== osd.7 v2:172.21.15.159:6816/3107756136 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd25c002dd0 con 0x7fd260073e00 2024-03-19T02:55:03.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.808+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 --> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d6015bb90 con 0x7f2d40073f50 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7f2d5cc00640 1 -- 172.21.15.92:0/1802354240 <== osd.5 v2:172.21.15.159:6824/1484023345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2d6015bb90 con 0x7f2d40073f50 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7fd260073e00 msgr2=0x7fd260076220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7fd288400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7fd260073e00 0x7fd260076220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd26006bc00 msgr2=0x7fd26006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7fd288400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd26006bc00 0x7fd26006e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009630 tx=0x7fd26c03b040 comp rx=0 tx=0).stop 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.809+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2800cbaa0 msgr2=0x7fd2801567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7fd288400640 1 --2- 172.21.15.92:0/1665602631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2800cbaa0 0x7fd2801567b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd27400e900 tx=0x7fd27400edc0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.841 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:03.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T02:55:03.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f2d40073f50 msgr2=0x7f2d40076370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7f2d66000640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f2d40073f50 0x7f2d40076370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7fd286400640 1 -- 172.21.15.92:0/1665602631 reap_dead start 2024-03-19T02:55:03.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 shutdown_connections 2024-03-19T02:55:03.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 >> 172.21.15.92:0/1665602631 conn(0x7fd28006da80 msgr2=0x7fd280066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.810+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2d4006bcd0 msgr2=0x7f2d4006e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 shutdown_connections 2024-03-19T02:55:03.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7fd288400640 1 -- 172.21.15.92:0/1665602631 wait complete. 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7f2d66000640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2d4006bcd0 0x7f2d4006e190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d58001dc0 tx=0x7f2d58009040 comp rx=0 tx=0).stop 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 msgr2=0x7f2d60200b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7f2d66000640 1 --2- 172.21.15.92:0/1802354240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d600cd3a0 0x7f2d60200b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e7f0 tx=0x7f2d5000ebf0 comp rx=0 tx=0).stop 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.811+0000 7f2d65600640 1 -- 172.21.15.92:0/1802354240 reap_dead start 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.812+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 shutdown_connections 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.812+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 >> 172.21.15.92:0/1802354240 conn(0x7f2d60066f50 msgr2=0x7f2d6015adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.812+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 shutdown_connections 2024-03-19T02:55:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:03.812+0000 7f2d66000640 1 -- 172.21.15.92:0/1802354240 wait complete. 2024-03-19T02:55:03.845 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:03.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T02:55:03.846 INFO:teuthology.orchestra.run.smithi092.stdout:55834574852 2024-03-19T02:55:03.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T02:55:03.847 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:03.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T02:55:03.848 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2024-03-19T02:55:03.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T02:55:03.849 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2024-03-19T02:55:03.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T02:55:03.850 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:03.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T02:55:03.851 INFO:teuthology.orchestra.run.smithi092.stdout:55834574852 2024-03-19T02:55:03.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T02:55:04.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.165+0000 7f618f200640 1 Processor -- start 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.166+0000 7f618f200640 1 -- start start 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618f200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f61800d9250 0x7f6180039e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618f200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f61800e0490 0x7f618004ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618f200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618003a4b0 con 0x7f61800dfaa0 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618f200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6180050510 con 0x7f61800e0490 2024-03-19T02:55:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618f200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f618004ef30 con 0x7f61800d9250 2024-03-19T02:55:04.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618d200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f61800dfaa0 0x7f61800071b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46906/0 (socket says 172.21.15.92:46906) 2024-03-19T02:55:04.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.168+0000 7f618d200640 1 -- 172.21.15.92:0/591906131 learned_addr learned my addr 172.21.15.92:0/591906131 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:04.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.169+0000 7f6187e00640 1 --2- 172.21.15.92:0/591906131 >> v2:172.21.15.159:3300/0 conn(0x7f61800d9250 0x7f6180039e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.169+0000 7f6187400640 1 --2- 172.21.15.92:0/591906131 >> v2:172.21.15.92:3300/0 conn(0x7f61800e0490 0x7f618004ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.169+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175724802 0 0) 0x7f618003a4b0 con 0x7f61800dfaa0 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.169+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6170003520 con 0x7f61800dfaa0 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.170+0000 7f6187400640 1 -- 172.21.15.92:0/591906131 >> v2:172.21.15.159:3300/0 conn(0x7f61800d9250 msgr2=0x7f6180039e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.170+0000 7f6187400640 1 --2- 172.21.15.92:0/591906131 >> v2:172.21.15.159:3300/0 conn(0x7f61800d9250 0x7f6180039e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.170+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4242570729 0 0) 0x7f6170003520 con 0x7f61800dfaa0 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.170+0000 7f6187400640 1 -- 172.21.15.92:0/591906131 >> v1:172.21.15.159:6789/0 conn(0x7f61800dfaa0 legacy=0x7f61800071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.170+0000 7f6187400640 1 -- 172.21.15.92:0/591906131 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618004f210 con 0x7f61800e0490 2024-03-19T02:55:04.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.171+0000 7f6187400640 1 --2- 172.21.15.92:0/591906131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f618004ffd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6180012090 tx=0x7f617c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427294a3d73134d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.172+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c038070 con 0x7f61800e0490 2024-03-19T02:55:04.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.172+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617c031500 con 0x7f61800e0490 2024-03-19T02:55:04.174 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.173+0000 7f618f200640 1 -- 172.21.15.92:0/591906131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 msgr2=0x7f618004ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.174 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.173+0000 7f6186a00640 1 -- 172.21.15.92:0/591906131 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c038070 con 0x7f61800e0490 2024-03-19T02:55:04.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.173+0000 7f618f200640 1 --2- 172.21.15.92:0/591906131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f618004ffd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6180012090 tx=0x7f617c02b820 comp rx=0 tx=0).stop 2024-03-19T02:55:04.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.174+0000 7f618f200640 1 -- 172.21.15.92:0/591906131 shutdown_connections 2024-03-19T02:55:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.174+0000 7f618f200640 1 --2- 172.21.15.92:0/591906131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f618004ffd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.174+0000 7f618f200640 1 --2- 172.21.15.92:0/591906131 >> v2:172.21.15.159:3300/0 conn(0x7f61800d9250 0x7f6180039e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.175+0000 7f618f200640 1 -- 172.21.15.92:0/591906131 >> 172.21.15.92:0/591906131 conn(0x7f618000e160 msgr2=0x7f618000e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.175+0000 7f618f200640 1 -- 172.21.15.92:0/591906131 shutdown_connections 2024-03-19T02:55:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.175+0000 7f618f200640 1 -- 172.21.15.92:0/591906131 wait complete. 2024-03-19T02:55:04.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.178+0000 7f618f200640 1 Processor -- start 2024-03-19T02:55:04.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.178+0000 7f618f200640 1 -- start start 2024-03-19T02:55:04.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.180+0000 7f4b19400640 1 Processor -- start 2024-03-19T02:55:04.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.180+0000 7f618f200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 0x7f61801755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.180+0000 7f618f200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 0x7f61800dcc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.181+0000 7f618f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f61800de060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.181+0000 7f6187400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 0x7f61800dcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.181+0000 7f6187400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 0x7f61800dcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60922/0 (socket says 172.21.15.92:60922) 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.181+0000 7f618d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f61800de060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f618d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f61800de060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48008/0 (socket says 172.21.15.92:48008) 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f6187400640 1 -- 172.21.15.92:0/2245073763 learned_addr learned my addr 172.21.15.92:0/2245073763 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7fd080000640 1 Processor -- start 2024-03-19T02:55:04.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f618004fab0 con 0x7f61800e0490 2024-03-19T02:55:04.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f6187e00640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 0x7f61801755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f618004f7d0 con 0x7f61800dfaa0 2024-03-19T02:55:04.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f61800dfef0 con 0x7f61800d9250 2024-03-19T02:55:04.184 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.180+0000 7f4b19400640 1 -- start start 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f4b19400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4b1406cea0 0x7f4b14157480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.182+0000 7f4b19400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140732f0 0x7f4b140c7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7fd080000640 1 -- start start 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b13e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4b1406cea0 0x7f4b14157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b13e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4b1406cea0 0x7f4b14157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60936/0 (socket says 172.21.15.92:60936) 2024-03-19T02:55:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b13e00640 1 -- 172.21.15.92:0/3584658005 learned_addr learned my addr 172.21.15.92:0/3584658005 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b13400640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140732f0 0x7f4b140c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f6187e00640 1 -- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 msgr2=0x7f61800dcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b140c7960 con 0x7f4b1406d280 2024-03-19T02:55:04.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f4b1406ea30 con 0x7f4b1406cea0 2024-03-19T02:55:04.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4b140c7c20 con 0x7f4b140732f0 2024-03-19T02:55:04.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f6187e00640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 0x7f61800dcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f6187e00640 1 -- 172.21.15.92:0/2245073763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 msgr2=0x7f61800de060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f6187e00640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f61800de060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.183+0000 7f6187e00640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6180174b30 con 0x7f61800d9250 2024-03-19T02:55:04.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.184+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 973296107 0 0) 0x7f4b140c7960 con 0x7f4b1406d280 2024-03-19T02:55:04.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.184+0000 7f6187e00640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 0x7f61801755a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f617400e8a0 tx=0x7f617400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.184+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4af8003520 con 0x7f4b1406d280 2024-03-19T02:55:04.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.184+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174017070 con 0x7f61800d9250 2024-03-19T02:55:04.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61800de7a0 con 0x7f61800d9250 2024-03-19T02:55:04.188 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6180174da0 con 0x7f61800d9250 2024-03-19T02:55:04.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617400c940 con 0x7f61800d9250 2024-03-19T02:55:04.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174009e60 con 0x7f61800d9250 2024-03-19T02:55:04.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7fd080000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd078154cd0 0x7fd07815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.186+0000 7fd080000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd078067ad0 0x7fd0780cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2323659493 0 0) 0x7f4af8003520 con 0x7f4b1406d280 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140732f0 msgr2=0x7f4b140c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140732f0 0x7f4b140c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 >> v2:172.21.15.159:3300/0 conn(0x7f4b1406cea0 msgr2=0x7f4b14157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 --2- 172.21.15.92:0/3584658005 >> v2:172.21.15.159:3300/0 conn(0x7f4b1406cea0 0x7f4b14157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1406ec80 con 0x7f4b1406d280 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7f4b0c0019f0 con 0x7f4b1406d280 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 >> v1:172.21.15.159:6789/0 conn(0x7f4b1406d280 legacy=0x7f4b14071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af80039b0 0x7f4af8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af8004800 0x7f4af8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af80099b0 0x7f4af80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1406ea30 con 0x7f4af80099b0 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4b140c7c20 con 0x7f4af80039b0 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.185+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4af8009da0 con 0x7f4af8004800 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.186+0000 7f4b18a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af80099b0 0x7f4af80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.187+0000 7f4b13e00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af80039b0 0x7f4af8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b13400640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af8004800 0x7f4af8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.186+0000 7fd080000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0780caf90 con 0x7fd0781550b0 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.186+0000 7fd07cc00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd0781550b0 0x7fd0780cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57970/0 (socket says 172.21.15.92:57970) 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.186+0000 7fd07cc00640 1 -- 172.21.15.92:0/720988446 learned_addr learned my addr 172.21.15.92:0/720988446 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.187+0000 7fd07d600640 1 --2- 172.21.15.92:0/720988446 >> v2:172.21.15.159:3300/0 conn(0x7fd078154cd0 0x7fd07815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.189+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd07815e280 con 0x7fd078067ad0 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.189+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd07815e520 con 0x7fd078154cd0 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b18a00640 1 -- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af8004800 msgr2=0x7f4af8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b18a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4af8004800 0x7f4af8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b18a00640 1 -- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af80039b0 msgr2=0x7f4af8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b18a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4af80039b0 0x7f4af8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.188+0000 7f4b18a00640 1 -- 172.21.15.92:0/3584658005 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1406ec80 con 0x7f4af80099b0 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.189+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536533075 0 0) 0x7fd06c001110 con 0x7fd0781550b0 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.189+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd060003520 con 0x7fd0781550b0 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2781918645 0 0) 0x7fd060003520 con 0x7fd0781550b0 2024-03-19T02:55:04.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 >> v2:172.21.15.159:3300/0 conn(0x7fd078154cd0 msgr2=0x7fd07815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 --2- 172.21.15.92:0/720988446 >> v2:172.21.15.159:3300/0 conn(0x7fd078154cd0 0x7fd07815ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 >> v2:172.21.15.92:3300/0 conn(0x7fd078067ad0 msgr2=0x7fd0780cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd07e000640 1 --2- 172.21.15.92:0/720988446 >> v2:172.21.15.92:3300/0 conn(0x7fd078067ad0 0x7fd0780cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.189+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164005290 con 0x7f61800d9250 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 --2- 172.21.15.92:0/720988446 >> v2:172.21.15.92:3300/0 conn(0x7fd078067ad0 0x7fd0780cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07815e800 con 0x7fd0781550b0 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1394095106 0 0) 0x7fd06c001840 con 0x7fd0781550b0 2024-03-19T02:55:04.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 >> v1:172.21.15.92:6789/0 conn(0x7fd0781550b0 legacy=0x7fd0780cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 0x7fd060003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7fd077e00640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060004800 0x7fd060004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd077e00640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600051c0 0x7fd0600141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd07815e280 con 0x7fd060004800 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd07815e520 con 0x7fd0600051c0 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd0600149a0 con 0x7fd0600039b0 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7f4b18a00640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af80099b0 0x7f4af80051c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4b1406d690 tx=0x7f4af8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3af31f2f7367e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c00c160 con 0x7f4af80099b0 2024-03-19T02:55:04.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7f4b12a00640 1 -- 172.21.15.92:0/3584658005 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0c02cdf0 con 0x7f4af80099b0 2024-03-19T02:55:04.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd07e000640 1 --2- 172.21.15.92:0/720988446 >> v2:172.21.15.92:3300/0 conn(0x7fd078067ad0 0x7fd0780cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:04.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd07d600640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 0x7fd060003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.191+0000 7fd07e000640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600051c0 0x7fd0600141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.190+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f617401a070 con 0x7f61800d9250 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07cc00640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060004800 0x7fd060004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af80099b0 msgr2=0x7f4af80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7f4b19400640 1 --2- 172.21.15.92:0/3584658005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af80099b0 0x7f4af80051c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4b1406d690 tx=0x7f4af8005830 comp rx=0 tx=0).stop 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7f4b18a00640 1 -- 172.21.15.92:0/3584658005 reap_dead start 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07d600640 1 -- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600051c0 msgr2=0x7fd0600141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 shutdown_connections 2024-03-19T02:55:04.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07d600640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600051c0 0x7fd0600141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.193+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 >> 172.21.15.92:0/3584658005 conn(0x7f4b1406bac0 msgr2=0x7f4b140cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07d600640 1 -- 172.21.15.92:0/720988446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060004800 msgr2=0x7fd060004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07d600640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060004800 0x7fd060004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.192+0000 7fd07d600640 1 -- 172.21.15.92:0/720988446 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07815e800 con 0x7fd0600039b0 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.193+0000 7fd07e000640 1 -- 172.21.15.92:0/720988446 reap_dead start 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.193+0000 7f617be00640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 0x7f616806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.194+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f617408abf0 con 0x7f61800d9250 2024-03-19T02:55:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.194+0000 7fd07d600640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 0x7fd060003dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd06400ba30 tx=0x7fd060005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9add6bdc12352bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.194+0000 7f6187400640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 0x7f616806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.194+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 shutdown_connections 2024-03-19T02:55:04.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.195+0000 7f74f6a00640 1 Processor -- start 2024-03-19T02:55:04.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.194+0000 7f4b19400640 1 -- 172.21.15.92:0/3584658005 wait complete. 2024-03-19T02:55:04.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.195+0000 7f38c2200640 1 Processor -- start 2024-03-19T02:55:04.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.196+0000 7f38c2200640 1 -- start start 2024-03-19T02:55:04.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.196+0000 7f6187400640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 0x7f616806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6180172650 tx=0x7f617c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f4b19400640 1 Processor -- start 2024-03-19T02:55:04.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.195+0000 7f74f6a00640 1 -- start start 2024-03-19T02:55:04.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74f6a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f74f0154ec0 0x7f74f015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74f6a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f74f00cdcc0 0x7f74f006cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74f6a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74f015e480 con 0x7f74f006d100 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74f6a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f74f015f000 con 0x7f74f00cdcc0 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74f6a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f74f01552a0 con 0x7f74f0154ec0 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74efe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f74f0154ec0 0x7f74f015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74efe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f74f0154ec0 0x7f74f015ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60966/0 (socket says 172.21.15.92:60966) 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.197+0000 7f74efe00640 1 -- 172.21.15.92:0/2806467889 learned_addr learned my addr 172.21.15.92:0/2806467889 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.198+0000 7f4b19400640 1 -- start start 2024-03-19T02:55:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.198+0000 7f38c2200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38c2200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f38bc071ec0 0x7f38bc15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6174058f60 con 0x7f61800d9250 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 0x7f4b141559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 0x7f4b141ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 0x7f4b141489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1406f760 con 0x7f4b1414ef40 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1406f240 con 0x7f4b1414f320 2024-03-19T02:55:04.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f4b19400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1406f8e0 con 0x7f4b140ca170 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38c2200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38bc0c6710 con 0x7f38bc154c90 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38c2200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f38bc0c6d70 con 0x7f38bc071ec0 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38c2200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f38bc072550 con 0x7f38bc06bc40 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38baa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38baa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37830/0 (socket says 172.21.15.92:37830) 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38bb400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f38bc154c90 0x7f38bc0cc140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58002/0 (socket says 172.21.15.92:58002) 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f38baa00640 1 -- 172.21.15.92:0/1228698440 learned_addr learned my addr 172.21.15.92:0/1228698440 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f0438e00640 1 Processor -- start 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f4b13400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 0x7f4b141ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f4b18a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 0x7f4b141489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f4b18a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 0x7f4b141489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37844/0 (socket says 172.21.15.92:37844) 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f4b18a00640 1 -- 172.21.15.92:0/3850365543 learned_addr learned my addr 172.21.15.92:0/3850365543 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.199+0000 7f74ef400640 1 --2- 172.21.15.92:0/2806467889 >> v2:172.21.15.92:3300/0 conn(0x7f74f00cdcc0 0x7f74f006cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f0438e00640 1 -- start start 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064036020 con 0x7fd0600039b0 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd064032660 con 0x7fd0600039b0 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7fd077e00640 1 -- 172.21.15.92:0/720988446 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06400b600 con 0x7fd0600039b0 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901831510 0 0) 0x7f74f015e480 con 0x7f74f006d100 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74d4003520 con 0x7f74f006d100 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f74efe00640 1 -- 172.21.15.92:0/2806467889 >> v2:172.21.15.92:3300/0 conn(0x7f74f00cdcc0 msgr2=0x7f74f006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 526254394 0 0) 0x7f74d4003520 con 0x7f74f006d100 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38ba000640 1 -- 172.21.15.92:0/1228698440 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214552985 0 0) 0x7f38bc0c6710 con 0x7f38bc154c90 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38ba000640 1 -- 172.21.15.92:0/1228698440 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38a0003520 con 0x7f38bc154c90 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.200+0000 7f4b13e00640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 0x7f4b141559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f74efe00640 1 --2- 172.21.15.92:0/2806467889 >> v2:172.21.15.92:3300/0 conn(0x7f74f00cdcc0 0x7f74f006cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f74efe00640 1 -- 172.21.15.92:0/2806467889 >> v1:172.21.15.92:6789/0 conn(0x7f74f006d100 legacy=0x7f74f00cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.208 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f74efe00640 1 -- 172.21.15.92:0/2806467889 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f015e760 con 0x7f74f0154ec0 2024-03-19T02:55:04.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 msgr2=0x7fd060003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7fd080000640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 0x7fd060003dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd06400ba30 tx=0x7fd060005830 comp rx=0 tx=0).stop 2024-03-19T02:55:04.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f1939e00640 1 Processor -- start 2024-03-19T02:55:04.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f0438e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 0x7f043415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f1939e00640 1 -- start start 2024-03-19T02:55:04.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 shutdown_connections 2024-03-19T02:55:04.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38baa00640 1 -- 172.21.15.92:0/1228698440 >> v2:172.21.15.92:3300/0 conn(0x7f38bc071ec0 msgr2=0x7f38bc15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38baa00640 1 --2- 172.21.15.92:0/1228698440 >> v2:172.21.15.92:3300/0 conn(0x7f38bc071ec0 0x7f38bc15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38baa00640 1 -- 172.21.15.92:0/1228698440 >> v1:172.21.15.92:6789/0 conn(0x7f38bc154c90 legacy=0x7f38bc0cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.201+0000 7f38baa00640 1 -- 172.21.15.92:0/1228698440 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38bc0c7050 con 0x7f38bc06bc40 2024-03-19T02:55:04.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f0431e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 0x7f043415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7f0431e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 0x7f043415ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:60980/0 (socket says 172.21.15.92:60980) 2024-03-19T02:55:04.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7f0431e00640 1 -- 172.21.15.92:0/2105139217 learned_addr learned my addr 172.21.15.92:0/2105139217 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.211 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f4b13400640 1 -- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 msgr2=0x7f4b141489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f4b13400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 0x7f4b141489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f4b13400640 1 -- 172.21.15.92:0/3850365543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 msgr2=0x7f4b141559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f4b13400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 0x7f4b141559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f4b13400640 1 -- 172.21.15.92:0/3850365543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1415aaf0 con 0x7f4b1414f320 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7f4b13400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 0x7f4b141ef200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b000107c0 tx=0x7f4b00010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f74efe00640 1 --2- 172.21.15.92:0/2806467889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f0154ec0 0x7f74f015ab50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f74f00cd7a0 tx=0x7f74e802b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bf2a44efe38aca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8038070 con 0x7f74f0154ec0 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e802f050 con 0x7f74f0154ec0 2024-03-19T02:55:04.212 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7f74eea00640 1 -- 172.21.15.92:0/2806467889 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00019f0 con 0x7f74f0154ec0 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f0438e00640 1 --2- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 0x7f04340cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b13e00640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 0x7f4b141559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0000f040 con 0x7f4b1414f320 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0000e950 con 0x7f4b1414f320 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0000ed20 con 0x7f4b1414f320 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1415ad90 con 0x7f4b1414f320 2024-03-19T02:55:04.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04340caf90 con 0x7f04341550b0 2024-03-19T02:55:04.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7fd080000640 1 --2- 172.21.15.92:0/720988446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0600039b0 0x7fd060003dd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.202+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 >> 172.21.15.92:0/720988446 conn(0x7fd078072970 msgr2=0x7fd078066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 shutdown_connections 2024-03-19T02:55:04.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.203+0000 7fd080000640 1 -- 172.21.15.92:0/720988446 wait complete. 2024-03-19T02:55:04.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b141ec040 con 0x7f4b1414f320 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f043415e280 con 0x7f0434154cd0 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f043415e520 con 0x7f0434067ad0 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f0432800640 1 --2- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 0x7f04340cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1934154ec0 0x7f193415ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f19340cdcc0 0x7f193406cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f193415f000 con 0x7f1934154ec0 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f19341552a0 con 0x7f19340cdcc0 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.204+0000 7f1939e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f193415e480 con 0x7f193406d100 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1934154ec0 0x7f193415ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1934154ec0 0x7f193415ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48046/0 (socket says 172.21.15.92:48046) 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 -- 172.21.15.92:0/2770694413 learned_addr learned my addr 172.21.15.92:0/2770694413 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 -- 172.21.15.92:0/2770694413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 msgr2=0x7f19340cc230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 --2- 172.21.15.92:0/2770694413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340cc230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 -- 172.21.15.92:0/2770694413 >> v2:172.21.15.159:3300/0 conn(0x7f19340cdcc0 msgr2=0x7f193406cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 --2- 172.21.15.92:0/2770694413 >> v2:172.21.15.159:3300/0 conn(0x7f19340cdcc0 0x7f193406cbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7f1932a00640 1 -- 172.21.15.92:0/2770694413 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193415e760 con 0x7f1934154ec0 2024-03-19T02:55:04.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.206+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1404f580 con 0x7f4b1414f320 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f1932a00640 1 --2- 172.21.15.92:0/2770694413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f193415ab50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f193406e2e0 tx=0x7f192c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65de1b3c9a2585b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.205+0000 7fd080000640 1 Processor -- start 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.206+0000 7fd080000640 1 -- start start 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 0x7fd07814b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 0x7fd07814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 0x7fd07814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0780cb1f0 con 0x7fd0780b3920 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd07815edc0 con 0x7fd0600039b0 2024-03-19T02:55:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7fd080000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd0780cb370 con 0x7fd0780b3d80 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 0x7fd07814ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 0x7fd07814ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37870/0 (socket says 172.21.15.92:37870) 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07e000640 1 -- 172.21.15.92:0/1108055347 learned_addr learned my addr 172.21.15.92:0/1108055347 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07cc00640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 0x7fd07814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07d600640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 0x7fd07814b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f74f6a00640 1 -- 172.21.15.92:0/2806467889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f0154ec0 msgr2=0x7f74f015ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2806467889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f0154ec0 0x7f74f015ab50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f74f00cd7a0 tx=0x7f74e802b870 comp rx=0 tx=0).stop 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f74f6a00640 1 -- 172.21.15.92:0/2806467889 shutdown_connections 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2806467889 >> v2:172.21.15.92:3300/0 conn(0x7f74f00cdcc0 0x7f74f006cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2806467889 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f0154ec0 0x7f74f015ab50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7f74f6a00640 1 -- 172.21.15.92:0/2806467889 >> 172.21.15.92:0/2806467889 conn(0x7f74f006da80 msgr2=0x7f74f006c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7f38baa00640 1 --2- 172.21.15.92:0/1228698440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009870 tx=0x7f38a402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dced4c134aa4dba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f0432800640 1 -- 172.21.15.92:0/2105139217 >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 msgr2=0x7f043415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f0432800640 1 --2- 172.21.15.92:0/2105139217 >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 0x7f043415ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f0432800640 1 -- 172.21.15.92:0/2105139217 >> v1:172.21.15.92:6789/0 conn(0x7f04341550b0 legacy=0x7f04340cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f0430a00640 1 -- 172.21.15.92:0/2105139217 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960986431 0 0) 0x7f0420001110 con 0x7f04341550b0 2024-03-19T02:55:04.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f0432800640 1 -- 172.21.15.92:0/2105139217 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043415e800 con 0x7f0434067ad0 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f1931600640 1 -- 172.21.15.92:0/2770694413 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c02cac0 con 0x7f1934154ec0 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f1931600640 1 -- 172.21.15.92:0/2770694413 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c02cc60 con 0x7f1934154ec0 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.207+0000 7f1931600640 1 -- 172.21.15.92:0/2770694413 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c02cac0 con 0x7f1934154ec0 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 -- 172.21.15.92:0/2770694413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 msgr2=0x7f193415ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 --2- 172.21.15.92:0/2770694413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f193415ab50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f193406e2e0 tx=0x7f192c02bf20 comp rx=0 tx=0).stop 2024-03-19T02:55:04.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f74f6a00640 1 -- 172.21.15.92:0/2806467889 shutdown_connections 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f38ba000640 1 -- 172.21.15.92:0/1228698440 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4035020 con 0x7f38bc06bc40 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f38ba000640 1 -- 172.21.15.92:0/1228698440 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a4031640 con 0x7f38bc06bc40 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.208+0000 7fd07d600640 1 -- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 msgr2=0x7fd07814ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7fd07d600640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 0x7fd07814ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7fd07d600640 1 -- 172.21.15.92:0/1108055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 msgr2=0x7fd07814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7fd07d600640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 0x7fd07814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f38ba000640 1 -- 172.21.15.92:0/1228698440 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a402d040 con 0x7f38bc06bc40 2024-03-19T02:55:04.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7fd07d600640 1 -- 172.21.15.92:0/1108055347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0780d0230 con 0x7fd0600039b0 2024-03-19T02:55:04.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f74f6a00640 1 -- 172.21.15.92:0/2806467889 wait complete. 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 -- 172.21.15.92:0/2770694413 shutdown_connections 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 --2- 172.21.15.92:0/2770694413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 --2- 172.21.15.92:0/2770694413 >> v2:172.21.15.159:3300/0 conn(0x7f19340cdcc0 0x7f193406cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 --2- 172.21.15.92:0/2770694413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f193415ab50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f1939e00640 1 -- 172.21.15.92:0/2770694413 >> 172.21.15.92:0/2770694413 conn(0x7f193406da80 msgr2=0x7f193406c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7fd07cc00640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 0x7fd07814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7fd07d600640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 0x7fd07814b540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4b0001c400 con 0x7f4b1414f320 2024-03-19T02:55:04.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f1939e00640 1 -- 172.21.15.92:0/2770694413 shutdown_connections 2024-03-19T02:55:04.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 -- 172.21.15.92:0/1228698440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 msgr2=0x7f38bc0c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 --2- 172.21.15.92:0/1228698440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009870 tx=0x7f38a402b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 -- 172.21.15.92:0/1228698440 shutdown_connections 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 --2- 172.21.15.92:0/1228698440 >> v2:172.21.15.92:3300/0 conn(0x7f38bc071ec0 0x7f38bc15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 --2- 172.21.15.92:0/1228698440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc0c5eb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f38c2200640 1 -- 172.21.15.92:0/1228698440 >> 172.21.15.92:0/1228698440 conn(0x7f38bc06ce30 msgr2=0x7f38bc06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f1939e00640 1 -- 172.21.15.92:0/2770694413 wait complete. 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f38c2200640 1 -- 172.21.15.92:0/1228698440 shutdown_connections 2024-03-19T02:55:04.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f38c2200640 1 -- 172.21.15.92:0/1228698440 wait complete. 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f4b0be00640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 0x7f4af406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4b0008af50 con 0x7f4b1414f320 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.209+0000 7f0432800640 1 --2- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 0x7f04340cbfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f042c0097a0 tx=0x7f042c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ea68b0395479c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f0430a00640 1 -- 172.21.15.92:0/2105139217 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c035020 con 0x7f0434067ad0 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f0430a00640 1 -- 172.21.15.92:0/2105139217 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042c031810 con 0x7f0434067ad0 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f0430a00640 1 -- 172.21.15.92:0/2105139217 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c031be0 con 0x7f0434067ad0 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 msgr2=0x7f04340cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.211+0000 7f0438e00640 1 --2- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 0x7f04340cbfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f042c0097a0 tx=0x7f042c02b8b0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f7841400640 1 Processor -- start 2024-03-19T02:55:04.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.210+0000 7f7841400640 1 -- start start 2024-03-19T02:55:04.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.212+0000 7f4b13e00640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 0x7f4af406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f1939e00640 1 Processor -- start 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f38c2200640 1 Processor -- start 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 shutdown_connections 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 --2- 172.21.15.92:0/2105139217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434067ad0 0x7f04340cbfa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 --2- 172.21.15.92:0/2105139217 >> v2:172.21.15.159:3300/0 conn(0x7f0434154cd0 0x7f043415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 >> 172.21.15.92:0/2105139217 conn(0x7f0434072970 msgr2=0x7f0434066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f1939e00640 1 -- start start 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.212+0000 7f7841400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f783c167930 0x7f783c0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.212+0000 7f7841400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c0c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.212+0000 7f7841400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783c072060 con 0x7f783c0c85a0 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f7841400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f783c071b30 con 0x7f783c167930 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f7841400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f783c072340 con 0x7f783c1651e0 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f783a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f783c167930 0x7f783c0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f783a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f783c167930 0x7f783c0c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48064/0 (socket says 172.21.15.92:48064) 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f783a800640 1 -- 172.21.15.92:0/1183569080 learned_addr learned my addr 172.21.15.92:0/1183569080 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f783a800640 1 -- 172.21.15.92:0/1183569080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 msgr2=0x7f783c0c6ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.213+0000 7f783b200640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c0c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f783a800640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c0c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f783a800640 1 -- 172.21.15.92:0/1183569080 >> v1:172.21.15.159:6789/0 conn(0x7f783c0c85a0 legacy=0x7f783c164ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:55:04.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f783a800640 1 -- 172.21.15.92:0/1183569080 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c072620 con 0x7f783c167930 2024-03-19T02:55:04.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f783b200640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c0c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:04.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 shutdown_connections 2024-03-19T02:55:04.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f0438e00640 1 -- 172.21.15.92:0/2105139217 wait complete. 2024-03-19T02:55:04.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f74f6a00640 1 Processor -- start 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19340cdcc0 0x7f19341e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f1934148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19340cbd10 con 0x7f1934154ec0 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f193415ed20 con 0x7f19340cdcc0 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1939e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f19340c7550 con 0x7f193406d100 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1932a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1932a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37892/0 (socket says 172.21.15.92:37892) 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1932a00640 1 -- 172.21.15.92:0/3002120253 learned_addr learned my addr 172.21.15.92:0/3002120253 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f1932000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19340cdcc0 0x7f19341e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064036040 con 0x7fd0600039b0 2024-03-19T02:55:04.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd064030a70 con 0x7fd0600039b0 2024-03-19T02:55:04.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06400b600 con 0x7fd0600039b0 2024-03-19T02:55:04.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0780cea20 con 0x7fd0600039b0 2024-03-19T02:55:04.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07814c4e0 con 0x7fd0600039b0 2024-03-19T02:55:04.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.214+0000 7f38c2200640 1 -- start start 2024-03-19T02:55:04.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f38c2200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f38c2200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f38bb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.215+0000 7f74f6a00640 1 -- start start 2024-03-19T02:55:04.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f38bb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37894/0 (socket says 172.21.15.92:37894) 2024-03-19T02:55:04.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f38baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f0438e00640 1 Processor -- start 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f38baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48092/0 (socket says 172.21.15.92:48092) 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f38c2200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38bc154c90 0x7f38bc0cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f38bb400640 1 -- 172.21.15.92:0/2797430044 learned_addr learned my addr 172.21.15.92:0/2797430044 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc0cc850 con 0x7f38bc071ec0 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc155070 con 0x7f38bc154c90 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f38bc0cc9d0 con 0x7f38bc06bc40 2024-03-19T02:55:04.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4b00018030 con 0x7f4b1414f320 2024-03-19T02:55:04.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0780cd1c0 con 0x7fd0600039b0 2024-03-19T02:55:04.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f38bbe00640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38bc154c90 0x7f38bc0cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f0438e00640 1 -- start start 2024-03-19T02:55:04.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f006d100 0x7f74f00d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f38baa00640 1 -- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 msgr2=0x7f38bc147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f38baa00640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f38baa00640 1 -- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38bc154c90 msgr2=0x7f38bc0cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f38baa00640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38bc154c90 0x7f38bc0cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.216+0000 7f1933400640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f1934148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f1932a00640 1 -- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19340cdcc0 msgr2=0x7f19341e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f1932a00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19340cdcc0 0x7f19341e5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f1932a00640 1 -- 172.21.15.92:0/3002120253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 msgr2=0x7f1934148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f1932a00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f1934148dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f1932a00640 1 -- 172.21.15.92:0/3002120253 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19341f0b90 con 0x7f193406d100 2024-03-19T02:55:04.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f1932a00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c009f40 tx=0x7f192c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.217+0000 7f783a800640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c167930 0x7f783c0c8060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f782c009870 tx=0x7f782c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0675fa6e3d6dd69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f38baa00640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38bc15e270 con 0x7f38bc071ec0 2024-03-19T02:55:04.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f4b13e00640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 0x7f4af406e0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b04006c50 tx=0x7f4b04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f7839400640 1 -- 172.21.15.92:0/1183569080 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c035020 con 0x7f783c167930 2024-03-19T02:55:04.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74efe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f006d100 0x7f74f00d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c042030 con 0x7f193406d100 2024-03-19T02:55:04.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c004680 con 0x7f193406d100 2024-03-19T02:55:04.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c009510 con 0x7f193406d100 2024-03-19T02:55:04.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 0x7f74f0159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.240 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74f00cbd10 con 0x7f74f00cdcc0 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74f015ed20 con 0x7f74f006d100 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74f6a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74f00c7550 con 0x7f74f0154ec0 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f74ef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48094/0 (socket says 172.21.15.92:48094) 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 -- 172.21.15.92:0/2547347696 learned_addr learned my addr 172.21.15.92:0/2547347696 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74f4a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 0x7f74f0159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 0x7f04340ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 0x7f04340ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f04340cd5c0 con 0x7f04341550b0 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f043415edc0 con 0x7f0434067ad0 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0438e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f04340cd740 con 0x7f0434154cd0 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 -- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 msgr2=0x7f74f0159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 0x7f74f0159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 -- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f006d100 msgr2=0x7f74f00d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f006d100 0x7f74f00d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74ef400640 1 -- 172.21.15.92:0/2547347696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f00cedf0 con 0x7f74f00cdcc0 2024-03-19T02:55:04.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f74f4a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 0x7f74f0159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0431e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0431e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:32810/0 (socket says 172.21.15.92:32810) 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f0431e00640 1 -- 172.21.15.92:0/2026306623 learned_addr learned my addr 172.21.15.92:0/2026306623 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f7839400640 1 -- 172.21.15.92:0/1183569080 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782c031640 con 0x7f783c167930 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.218+0000 7f7839400640 1 -- 172.21.15.92:0/1183569080 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c02d040 con 0x7f783c167930 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 -- 172.21.15.92:0/1183569080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c167930 msgr2=0x7f783c0c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c167930 0x7f783c0c8060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f782c009870 tx=0x7f782c02b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 -- 172.21.15.92:0/1183569080 shutdown_connections 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c0c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 --2- 172.21.15.92:0/1183569080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c167930 0x7f783c0c8060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f7841400640 1 -- 172.21.15.92:0/1183569080 >> 172.21.15.92:0/1183569080 conn(0x7f783c0c8d90 msgr2=0x7f783c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f7841400640 1 -- 172.21.15.92:0/1183569080 shutdown_connections 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0432800640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 0x7f04340ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19341f4ba0 con 0x7f193406d100 2024-03-19T02:55:04.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19341f1110 con 0x7f193406d100 2024-03-19T02:55:04.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f7841400640 1 -- 172.21.15.92:0/1183569080 wait complete. 2024-03-19T02:55:04.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.219+0000 7f38baa00640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f38a4033000 tx=0x7f38a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431e00640 1 -- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 msgr2=0x7f04340ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 0x7f04340ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431e00640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 0x7f04340ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431e00640 1 -- 172.21.15.92:0/2026306623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 msgr2=0x7f04340ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431e00640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 0x7f04340ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f0431e00640 1 -- 172.21.15.92:0/2026306623 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04340b55a0 con 0x7f0434067ad0 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f0431400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 0x7f04340ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:04.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f0432800640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 0x7f04340ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f74ef400640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74d800eb40 tx=0x7f74d800c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8017070 con 0x7f74f00cdcc0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f0158b30 con 0x7f74f00cdcc0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f00cf370 con 0x7f74f00cdcc0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d800c980 con 0x7f74f00cdcc0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d800cd50 con 0x7f74f00cdcc0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.222+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd064039020 con 0x7fd0600039b0 2024-03-19T02:55:04.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4040070 con 0x7f38bc071ec0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a40399f0 con 0x7f38bc071ec0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.220+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4035040 con 0x7f38bc071ec0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.222+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38bc15e510 con 0x7f38bc071ec0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.222+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38bc15ead0 con 0x7f38bc071ec0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.221+0000 7f0431e00640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0424009cf0 tx=0x7f042400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042400cda0 con 0x7f0434067ad0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0424019070 con 0x7f0434067ad0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04240146a0 con 0x7f0434067ad0 2024-03-19T02:55:04.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f0438e00640 1 -- 172.21.15.92:0/2026306623 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0434146e10 con 0x7f0434067ad0 2024-03-19T02:55:04.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7f7841400640 1 Processor -- start 2024-03-19T02:55:04.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7fd075600640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7f7841400640 1 -- start start 2024-03-19T02:55:04.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd0640ad2d0 con 0x7fd0600039b0 2024-03-19T02:55:04.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f0438e00640 1 -- 172.21.15.92:0/2026306623 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04341473d0 con 0x7f0434067ad0 2024-03-19T02:55:04.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7fd07cc00640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.224+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38bc04f580 con 0x7f38bc071ec0 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.225+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c8005290 con 0x7f74f00cdcc0 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.225+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0434146c00 con 0x7f0434067ad0 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.222+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19340684c0 con 0x7f193406d100 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.223+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f192c045070 con 0x7f193406d100 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.225+0000 7f192b400640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 0x7f191406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.225+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f192c0acb90 con 0x7f193406d100 2024-03-19T02:55:04.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.225+0000 7f1932000640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 0x7f191406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f38a404b060 con 0x7f38bc071ec0 2024-03-19T02:55:04.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7fd07cc00640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 0x7fd05806e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd078148f70 tx=0x7fd06c001600 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f7841400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f7841400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c1fde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f1932000640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 0x7f191406e160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f783a800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f783a800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:32812/0 (socket says 172.21.15.92:32812) 2024-03-19T02:55:04.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f7839e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c1fde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f7841400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c0cb550 0x7f783c0c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f7839e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c1fde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37928/0 (socket says 172.21.15.92:37928) 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f7841400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f783c0c3920 con 0x7f783c0cb550 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f7841400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f783c073300 con 0x7f783c0c85a0 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f7841400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f783c071190 con 0x7f783c1651e0 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 -- 172.21.15.92:0/2951128983 learned_addr learned my addr 172.21.15.92:0/2951128983 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783b200640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c0cb550 0x7f783c0c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f38b3400640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 0x7f389806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f38a40ac240 con 0x7f38bc071ec0 2024-03-19T02:55:04.255 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 -- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 msgr2=0x7f783c1fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.255 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c1fde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.255 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 -- 172.21.15.92:0/2951128983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c0cb550 msgr2=0x7f783c0c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.255 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c0cb550 0x7f783c0c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.255 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.227+0000 7f783a800640 1 -- 172.21.15.92:0/2951128983 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c0c5860 con 0x7f783c0c85a0 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.226+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f74d801a070 con 0x7f74f00cdcc0 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.228+0000 7f74e7e00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 0x7f74cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.228+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f74d808ae50 con 0x7f74f00cdcc0 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.228+0000 7f783a800640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f782c000c00 tx=0x7f782c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.228+0000 7f74efe00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 0x7f74cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.229+0000 7f74efe00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 0x7f74cc06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74e8035000 tx=0x7f74e8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.229+0000 7f38bb400640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 0x7f389806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.229+0000 7f38bb400640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 0x7f389806e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38b4007fd0 tx=0x7f38b4001780 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.232+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f38a407a640 con 0x7f38bc071ec0 2024-03-19T02:55:04.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.232+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c044070 con 0x7f783c0c85a0 2024-03-19T02:55:04.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.232+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782c03ed20 con 0x7f783c0c85a0 2024-03-19T02:55:04.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.232+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c035040 con 0x7f783c0c85a0 2024-03-19T02:55:04.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.233+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd0780cd1c0 con 0x7fd0600039b0 2024-03-19T02:55:04.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.232+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783c0c5b00 con 0x7f783c0c85a0 2024-03-19T02:55:04.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.233+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783c0c42a0 con 0x7f783c0c85a0 2024-03-19T02:55:04.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.234+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f192c04c070 con 0x7f193406d100 2024-03-19T02:55:04.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.234+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783c068de0 con 0x7f783c0c85a0 2024-03-19T02:55:04.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.239+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0424005d10 con 0x7f0434067ad0 2024-03-19T02:55:04.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.243+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f782c04f060 con 0x7f783c0c85a0 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.243+0000 7f042a000640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 0x7f041406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.244+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f042408bae0 con 0x7f0434067ad0 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.244+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0424010040 con 0x7f0434067ad0 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.245+0000 7f0431400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 0x7f041406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.245+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74d80591c0 con 0x7f74f00cdcc0 2024-03-19T02:55:04.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.245+0000 7f7832a00640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 0x7f781c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.245+0000 7f0431400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 0x7f041406e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f043406cf50 tx=0x7f0420009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.246+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f782c0b0b70 con 0x7f783c0c85a0 2024-03-19T02:55:04.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.246+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f783c068de0 con 0x7f783c0c85a0 2024-03-19T02:55:04.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.245+0000 7f7839e00640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 0x7f781c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.251+0000 7f7839e00640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 0x7f781c06e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f783c06f290 tx=0x7f78280090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.624+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6164005580 con 0x7f61800d9250 2024-03-19T02:55:04.629 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:04.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.625+0000 7f617be00640 1 -- 172.21.15.92:0/2245073763 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f617405d000 con 0x7f61800d9250 2024-03-19T02:55:04.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.629+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 msgr2=0x7f616806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.629+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 0x7f616806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6180172650 tx=0x7f617c008030 comp rx=0 tx=0).stop 2024-03-19T02:55:04.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.629+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 msgr2=0x7f61801755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.629+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 0x7f61801755a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f617400e8a0 tx=0x7f617400ed60 comp rx=0 tx=0).stop 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 shutdown_connections 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f616806bbd0 0x7f616806e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61800e0490 0x7f61800de060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800dfaa0 0x7f61800dcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 --2- 172.21.15.92:0/2245073763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61800d9250 0x7f61801755a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.630+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 >> 172.21.15.92:0/2245073763 conn(0x7f618000e160 msgr2=0x7f61800d7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.631+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 shutdown_connections 2024-03-19T02:55:04.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.631+0000 7f618f200640 1 -- 172.21.15.92:0/2245073763 wait complete. 2024-03-19T02:55:04.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.635+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4b1414e790 con 0x7f4b1414f320 2024-03-19T02:55:04.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.636+0000 7f4b0be00640 1 -- 172.21.15.92:0/3850365543 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4b00059a20 con 0x7f4b1414f320 2024-03-19T02:55:04.642 INFO:teuthology.orchestra.run.smithi092.stdout:55834574852 2024-03-19T02:55:04.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.641+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 msgr2=0x7f4af406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.641+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 0x7f4af406e0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b04006c50 tx=0x7f4b04008040 comp rx=0 tx=0).stop 2024-03-19T02:55:04.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.641+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 msgr2=0x7f4b141ef200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.641+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 0x7f4b141ef200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b000107c0 tx=0x7f4b00010c80 comp rx=0 tx=0).stop 2024-03-19T02:55:04.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 shutdown_connections 2024-03-19T02:55:04.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4af406bc20 0x7f4af406e0e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b140ca170 0x7f4b141489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b1414f320 0x7f4b141ef200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 --2- 172.21.15.92:0/3850365543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b1414ef40 0x7f4b141559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 >> 172.21.15.92:0/3850365543 conn(0x7f4b1406bac0 msgr2=0x7f4b141505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 shutdown_connections 2024-03-19T02:55:04.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.642+0000 7f4b19400640 1 -- 172.21.15.92:0/3850365543 wait complete. 2024-03-19T02:55:04.659 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-03-19T02:55:04.659 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.664+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f193414f300 con 0x7f193406d100 2024-03-19T02:55:04.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.664+0000 7f192b400640 1 -- 172.21.15.92:0/3002120253 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f192c07b6f0 con 0x7f193406d100 2024-03-19T02:55:04.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.665+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd0780684c0 con 0x7fd0600039b0 2024-03-19T02:55:04.666 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:04.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.665+0000 7fd075600640 1 -- 172.21.15.92:0/1108055347 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd06407beb0 con 0x7fd0600039b0 2024-03-19T02:55:04.667 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:04.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.665+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f74c8005580 con 0x7f74f00cdcc0 2024-03-19T02:55:04.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.666+0000 7f74e7e00640 1 -- 172.21.15.92:0/2547347696 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f74d808c090 con 0x7f74f00cdcc0 2024-03-19T02:55:04.668 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2024-03-19T02:55:04.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2024-03-19T02:55:04.669 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.668+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 msgr2=0x7f74cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.668+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 0x7f74cc06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74e8035000 tx=0x7f74e8008030 comp rx=0 tx=0).stop 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 msgr2=0x7f74f0159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74d800eb40 tx=0x7f74d800c150 comp rx=0 tx=0).stop 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 msgr2=0x7fd05806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 0x7fd05806e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd078148f70 tx=0x7fd06c001600 comp rx=0 tx=0).stop 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 msgr2=0x7fd07814b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 0x7fd07814b540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064004730 comp rx=0 tx=0).stop 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 msgr2=0x7f191406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 0x7f191406e160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924008040 comp rx=0 tx=0).stop 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 msgr2=0x7f19340671c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c009f40 tx=0x7f192c004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f04340684c0 con 0x7f0434067ad0 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 shutdown_connections 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74cc06bbd0 0x7f74cc06e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74f0154ec0 0x7f74f0159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00cdcc0 0x7f74f0159320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 --2- 172.21.15.92:0/2547347696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74f006d100 0x7f74f00d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 >> 172.21.15.92:0/2547347696 conn(0x7f74f006da80 msgr2=0x7f74f0153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 shutdown_connections 2024-03-19T02:55:04.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 shutdown_connections 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f191406bca0 0x7f191406e160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1934154ec0 0x7f1934148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19340cdcc0 0x7f19341e5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 --2- 172.21.15.92:0/3002120253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f193406d100 0x7f19340671c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 >> 172.21.15.92:0/3002120253 conn(0x7f193406da80 msgr2=0x7f19341500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 shutdown_connections 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 shutdown_connections 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0780b3d80 0x7fd07814ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780b3920 0x7fd07814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 --2- 172.21.15.92:0/1108055347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600039b0 0x7fd07814b540 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 >> 172.21.15.92:0/1108055347 conn(0x7fd078072970 msgr2=0x7fd0780c6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f1939e00640 1 -- 172.21.15.92:0/3002120253 wait complete. 2024-03-19T02:55:04.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f74f6a00640 1 -- 172.21.15.92:0/2547347696 wait complete. 2024-03-19T02:55:04.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 shutdown_connections 2024-03-19T02:55:04.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7f042a000640 1 -- 172.21.15.92:0/2026306623 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f042405a520 con 0x7f0434067ad0 2024-03-19T02:55:04.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.669+0000 7fd080000640 1 -- 172.21.15.92:0/1108055347 wait complete. 2024-03-19T02:55:04.676 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2024-03-19T02:55:04.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.671+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f38bc06c250 con 0x7f38bc071ec0 2024-03-19T02:55:04.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.672+0000 7f38b3400640 1 -- 172.21.15.92:0/2797430044 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f38a407e6e0 con 0x7f38bc071ec0 2024-03-19T02:55:04.676 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.672+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 msgr2=0x7f041406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.672+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 0x7f041406e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f043406cf50 tx=0x7f0420009040 comp rx=0 tx=0).stop 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.672+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 msgr2=0x7f043413c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.672+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0424009cf0 tx=0x7f042400c490 comp rx=0 tx=0).stop 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 shutdown_connections 2024-03-19T02:55:04.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041406bca0 0x7f041406e160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04341550b0 0x7f04340ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0434154cd0 0x7f04340ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 --2- 172.21.15.92:0/2026306623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0434067ad0 0x7f043413c820 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.673+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 >> 172.21.15.92:0/2026306623 conn(0x7f0434072970 msgr2=0x7f04340cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.675+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 shutdown_connections 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.675+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f783c0684c0 con 0x7f783c0c85a0 2024-03-19T02:55:04.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.675+0000 7f041f400640 1 -- 172.21.15.92:0/2026306623 wait complete. 2024-03-19T02:55:04.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.675+0000 7f7832a00640 1 -- 172.21.15.92:0/2951128983 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f782c031610 con 0x7f783c0c85a0 2024-03-19T02:55:04.679 INFO:teuthology.orchestra.run.smithi092.stdout:55834574852 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 msgr2=0x7f389806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 0x7f389806e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38b4007fd0 tx=0x7f38b4001780 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 msgr2=0x7f38bc158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f38a4033000 tx=0x7f38a40046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 shutdown_connections 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f389806bbd0 0x7f389806e090 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38b4007fd0 tx=0x7f38b4001780 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38bc154c90 0x7f38bc0cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38bc071ec0 0x7f38bc158790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 --2- 172.21.15.92:0/2797430044 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38bc06bc40 0x7f38bc147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 >> 172.21.15.92:0/2797430044 conn(0x7f38bc06ce30 msgr2=0x7f38bc06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 shutdown_connections 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.676+0000 7f38c2200640 1 -- 172.21.15.92:0/2797430044 wait complete. 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 msgr2=0x7f781c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 0x7f781c06e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f783c06f290 tx=0x7f78280090e0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 msgr2=0x7f783c15d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f782c000c00 tx=0x7f782c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 shutdown_connections 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f781c06bbd0 0x7f781c06e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f783c0cb550 0x7f783c0c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f783c1651e0 0x7f783c1fde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 --2- 172.21.15.92:0/2951128983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f783c0c85a0 0x7f783c15d810 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 >> 172.21.15.92:0/2951128983 conn(0x7f783c0c8d90 msgr2=0x7f783c163fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 shutdown_connections 2024-03-19T02:55:04.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.679+0000 7f7841400640 1 -- 172.21.15.92:0/2951128983 wait complete. 2024-03-19T02:55:04.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2024-03-19T02:55:04.700 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-03-19T02:55:04.700 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-03-19T02:55:04.701 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-19T02:55:04.703 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-19T02:55:04.705 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.706 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2024-03-19T02:55:04.706 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:04.706 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T02:55:04.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:04.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.941+0000 7f0ff5a00640 1 Processor -- start 2024-03-19T02:55:04.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.941+0000 7f0ff5a00640 1 -- start start 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0ff5a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0ff0154b10 0x7f0ff00cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0ff5a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00c63e0 0x7f0ff006c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0ff5a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff015b9e0 con 0x7f0ff015b550 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0ff5a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0ff00c5340 con 0x7f0ff0154b10 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0ff5a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff00c5620 con 0x7f0ff00c63e0 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0fe6000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f0ff015b550 0x7f0ff015c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46952/0 (socket says 172.21.15.92:46952) 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0fe6000640 1 -- 172.21.15.92:0/3747658263 learned_addr learned my addr 172.21.15.92:0/3747658263 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0feea00640 1 --2- 172.21.15.92:0/3747658263 >> v2:172.21.15.92:3300/0 conn(0x7f0ff0154b10 0x7f0ff00cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0fef400640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00c63e0 0x7f0ff006c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.943+0000 7f0fee000640 1 -- 172.21.15.92:0/3747658263 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446495136 0 0) 0x7f0ff015b9e0 con 0x7f0ff015b550 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.944+0000 7f0fee000640 1 -- 172.21.15.92:0/3747658263 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd4003520 con 0x7f0ff015b550 2024-03-19T02:55:04.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.944+0000 7f0feea00640 1 -- 172.21.15.92:0/3747658263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00c63e0 msgr2=0x7f0ff006c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:04.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.944+0000 7f0feea00640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00c63e0 0x7f0ff006c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.944+0000 7f0feea00640 1 -- 172.21.15.92:0/3747658263 >> v1:172.21.15.159:6789/0 conn(0x7f0ff015b550 legacy=0x7f0ff015c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.944+0000 7f0feea00640 1 -- 172.21.15.92:0/3747658263 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff00c5900 con 0x7f0ff0154b10 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0feea00640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff0154b10 0x7f0ff00cae40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00097a0 tx=0x7f0fe002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23aae705c39f3ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0fee000640 1 -- 172.21.15.92:0/3747658263 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0034070 con 0x7f0ff0154b10 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0fee000640 1 -- 172.21.15.92:0/3747658263 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe0030610 con 0x7f0ff0154b10 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0fee000640 1 -- 172.21.15.92:0/3747658263 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00309e0 con 0x7f0ff0154b10 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0ff5a00640 1 -- 172.21.15.92:0/3747658263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff0154b10 msgr2=0x7f0ff00cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.945+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff0154b10 0x7f0ff00cae40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00097a0 tx=0x7f0fe002b960 comp rx=0 tx=0).stop 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 -- 172.21.15.92:0/3747658263 shutdown_connections 2024-03-19T02:55:04.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00c63e0 0x7f0ff006c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/3747658263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff0154b10 0x7f0ff00cae40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 -- 172.21.15.92:0/3747658263 >> 172.21.15.92:0/3747658263 conn(0x7f0ff0072970 msgr2=0x7f0ff0066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 -- 172.21.15.92:0/3747658263 shutdown_connections 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.946+0000 7f0ff5a00640 1 -- 172.21.15.92:0/3747658263 wait complete. 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.947+0000 7f0ff5a00640 1 Processor -- start 2024-03-19T02:55:04.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.947+0000 7f0ff5a00640 1 -- start start 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ff0154b10 0x7f0ff01f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00ca4b0 0x7f0ff014dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0067ca0 con 0x7f0ff00c63e0 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff00c5ec0 con 0x7f0ff0154b10 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0ff5a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff0154ef0 con 0x7f0ff00ca4b0 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0feea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0feea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48128/0 (socket says 172.21.15.92:48128) 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.948+0000 7f0feea00640 1 -- 172.21.15.92:0/2676088896 learned_addr learned my addr 172.21.15.92:0/2676088896 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0fe6000640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ff0154b10 0x7f0ff01f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0fef400640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00ca4b0 0x7f0ff014dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 -- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00ca4b0 msgr2=0x7f0ff014dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00ca4b0 0x7f0ff014dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 -- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ff0154b10 msgr2=0x7f0ff01f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ff0154b10 0x7f0ff01f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 -- 172.21.15.92:0/2676088896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff014f1c0 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0feea00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0fe002bf70 tx=0x7f0fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0034040 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe0030410 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0006e10 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.949+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff01f14d0 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.950+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff00cd190 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.951+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff01f40d0 con 0x7f0ff00c63e0 2024-03-19T02:55:04.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.951+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0fe0037070 con 0x7f0ff00c63e0 2024-03-19T02:55:04.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.952+0000 7f0fe7400640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 0x7f0fcc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:04.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.952+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0fe00ac7c0 con 0x7f0ff00c63e0 2024-03-19T02:55:04.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.952+0000 7f0fe6000640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 0x7f0fcc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:04.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.953+0000 7f0fe6000640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 0x7f0fcc06e0c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0153e70 tx=0x7f0fd8007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:04.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:04.957+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0fe007ac50 con 0x7f0ff00c63e0 2024-03-19T02:55:05.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.138+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ff01510d0 con 0x7f0fcc06bc00 2024-03-19T02:55:05.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.146+0000 7f0fe7400640 1 -- 172.21.15.92:0/2676088896 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252024 (secure 0 0 0) 0x7f0ff01510d0 con 0x7f0fcc06bc00 2024-03-19T02:55:05.150 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:05.151 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 msgr2=0x7f0fcc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 0x7f0fcc06e0c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0153e70 tx=0x7f0fd8007d30 comp rx=0 tx=0).stop 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 msgr2=0x7f0ff00b5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0fe002bf70 tx=0x7f0fe00046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 shutdown_connections 2024-03-19T02:55:05.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0fcc06bc00 0x7f0fcc06e0c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ff00ca4b0 0x7f0ff014dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ff0154b10 0x7f0ff01f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 --2- 172.21.15.92:0/2676088896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff00c63e0 0x7f0ff00b5260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.150+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 >> 172.21.15.92:0/2676088896 conn(0x7f0ff0072970 msgr2=0x7f0ff0151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.151+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 shutdown_connections 2024-03-19T02:55:05.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.151+0000 7f0ff5a00640 1 -- 172.21.15.92:0/2676088896 wait complete. 2024-03-19T02:55:05.168 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-19T02:55:03.837574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217920,"kb_used_data":1792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756800,"statfs":{"total":773094113280,"available":772870963200,"internally_reserved":0,"allocated":1835008,"data_stored":1753028,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.663089"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380801+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380801+0000","last_peered":"2024-03-19T02:54:57.380801+0000","last_clean":"2024-03-19T02:54:57.380801+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380801+0000","last_undegraded":"2024-03-19T02:54:57.380801+0000","last_fullsized":"2024-03-19T02:54:57.380801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081567+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081567+0000","last_peered":"2024-03-19T02:54:59.081567+0000","last_clean":"2024-03-19T02:54:59.081567+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081567+0000","last_undegraded":"2024-03-19T02:54:59.081567+0000","last_fullsized":"2024-03-19T02:54:59.081567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.182131+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:55:01.182131+0000","last_peered":"2024-03-19T02:55:01.182131+0000","last_clean":"2024-03-19T02:55:01.182131+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:55:01.182131+0000","last_undegraded":"2024-03-19T02:55:01.182131+0000","last_fullsized":"2024-03-19T02:55:01.182131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183040+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:55:01.183040+0000","last_peered":"2024-03-19T02:55:01.183040+0000","last_clean":"2024-03-19T02:55:01.183040+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:55:01.183040+0000","last_undegraded":"2024-03-19T02:55:01.183040+0000","last_fullsized":"2024-03-19T02:55:01.183040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183309+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:55:01.183309+0000","last_peered":"2024-03-19T02:55:01.183309+0000","last_clean":"2024-03-19T02:55:01.183309+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:55:01.183309+0000","last_undegraded":"2024-03-19T02:55:01.183309+0000","last_fullsized":"2024-03-19T02:55:01.183309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.186421+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:55:01.186421+0000","last_peered":"2024-03-19T02:55:01.186421+0000","last_clean":"2024-03-19T02:55:01.186421+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:55:01.186421+0000","last_undegraded":"2024-03-19T02:55:01.186421+0000","last_fullsized":"2024-03-19T02:55:01.186421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197959+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197959+0000","last_peered":"2024-03-19T02:54:57.197959+0000","last_clean":"2024-03-19T02:54:57.197959+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197959+0000","last_undegraded":"2024-03-19T02:54:57.197959+0000","last_fullsized":"2024-03-19T02:54:57.197959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188015+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:55:01.188015+0000","last_peered":"2024-03-19T02:55:01.188015+0000","last_clean":"2024-03-19T02:55:01.188015+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:55:01.188015+0000","last_undegraded":"2024-03-19T02:55:01.188015+0000","last_fullsized":"2024-03-19T02:55:01.188015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201693+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201693+0000","last_peered":"2024-03-19T02:54:57.201693+0000","last_clean":"2024-03-19T02:54:57.201693+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201693+0000","last_undegraded":"2024-03-19T02:54:57.201693+0000","last_fullsized":"2024-03-19T02:54:57.201693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201716+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201716+0000","last_peered":"2024-03-19T02:54:57.201716+0000","last_clean":"2024-03-19T02:54:57.201716+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201716+0000","last_undegraded":"2024-03-19T02:54:57.201716+0000","last_fullsized":"2024-03-19T02:54:57.201716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177520+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:55:01.177520+0000","last_peered":"2024-03-19T02:55:01.177520+0000","last_clean":"2024-03-19T02:55:01.177520+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:55:01.177520+0000","last_undegraded":"2024-03-19T02:55:01.177520+0000","last_fullsized":"2024-03-19T02:55:01.177520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177763+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:55:01.177763+0000","last_peered":"2024-03-19T02:55:01.177763+0000","last_clean":"2024-03-19T02:55:01.177763+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:55:01.177763+0000","last_undegraded":"2024-03-19T02:55:01.177763+0000","last_fullsized":"2024-03-19T02:55:01.177763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177777+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:55:01.177777+0000","last_peered":"2024-03-19T02:55:01.177777+0000","last_clean":"2024-03-19T02:55:01.177777+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:55:01.177777+0000","last_undegraded":"2024-03-19T02:55:01.177777+0000","last_fullsized":"2024-03-19T02:55:01.177777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.178273+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:55:01.178273+0000","last_peered":"2024-03-19T02:55:01.178273+0000","last_clean":"2024-03-19T02:55:01.178273+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:55:01.178273+0000","last_undegraded":"2024-03-19T02:55:01.178273+0000","last_fullsized":"2024-03-19T02:55:01.178273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381552+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381552+0000","last_peered":"2024-03-19T02:54:57.381552+0000","last_clean":"2024-03-19T02:54:57.381552+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381552+0000","last_undegraded":"2024-03-19T02:54:57.381552+0000","last_fullsized":"2024-03-19T02:54:57.381552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188465+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:55:01.188465+0000","last_peered":"2024-03-19T02:55:01.188465+0000","last_clean":"2024-03-19T02:55:01.188465+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:55:01.188465+0000","last_undegraded":"2024-03-19T02:55:01.188465+0000","last_fullsized":"2024-03-19T02:55:01.188465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661884,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":661940,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:05.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:05.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.412+0000 7f2e52e00640 1 Processor -- start 2024-03-19T02:55:05.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.412+0000 7f2e52e00640 1 -- start start 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e52e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2e4c0c6340 0x7f2e4c0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e52e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4c157870 con 0x7f2e4c151ad0 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e52e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4c0c4a40 con 0x7f2e4c0c7c00 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e52e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2e4c158830 con 0x7f2e4c0c6340 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e4be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2e4c151ad0 0x7f2e4c072f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58014/0 (socket says 172.21.15.92:58014) 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e4be00640 1 -- 172.21.15.92:0/3250335917 learned_addr learned my addr 172.21.15.92:0/3250335917 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e50e00640 1 --2- 172.21.15.92:0/3250335917 >> v2:172.21.15.92:3300/0 conn(0x7f2e4c0c6340 0x7f2e4c0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.414+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699258302 0 0) 0x7f2e4c157870 con 0x7f2e4c151ad0 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e2c003520 con 0x7f2e4c151ad0 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435786149 0 0) 0x7f2e4c0c4a40 con 0x7f2e4c0c7c00 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e4c157870 con 0x7f2e4c0c7c00 2024-03-19T02:55:05.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e50e00640 1 -- 172.21.15.92:0/3250335917 >> v1:172.21.15.159:6789/0 conn(0x7f2e4c0c7c00 legacy=0x7f2e4c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e50e00640 1 -- 172.21.15.92:0/3250335917 >> v1:172.21.15.92:6789/0 conn(0x7f2e4c151ad0 legacy=0x7f2e4c072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.415+0000 7f2e50e00640 1 -- 172.21.15.92:0/3250335917 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4c0c94c0 con 0x7f2e4c0c6340 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e50e00640 1 --2- 172.21.15.92:0/3250335917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c0c6340 0x7f2e4c0c6740 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e440097a0 tx=0x7f2e4402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71320fc4d68c1a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44034070 con 0x7f2e4c0c6340 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e44030610 con 0x7f2e4c0c6340 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e4aa00640 1 -- 172.21.15.92:0/3250335917 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440309e0 con 0x7f2e4c0c6340 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 -- 172.21.15.92:0/3250335917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c0c6340 msgr2=0x7f2e4c0c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 --2- 172.21.15.92:0/3250335917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c0c6340 0x7f2e4c0c6740 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e440097a0 tx=0x7f2e4402b960 comp rx=0 tx=0).stop 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 -- 172.21.15.92:0/3250335917 shutdown_connections 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 --2- 172.21.15.92:0/3250335917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c0c6340 0x7f2e4c0c6740 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 -- 172.21.15.92:0/3250335917 >> 172.21.15.92:0/3250335917 conn(0x7f2e4c0cd200 msgr2=0x7f2e4c0cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.416+0000 7f2e52e00640 1 -- 172.21.15.92:0/3250335917 shutdown_connections 2024-03-19T02:55:05.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.417+0000 7f2e52e00640 1 -- 172.21.15.92:0/3250335917 wait complete. 2024-03-19T02:55:05.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.418+0000 7f2e52e00640 1 Processor -- start 2024-03-19T02:55:05.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.418+0000 7f2e52e00640 1 -- start start 2024-03-19T02:55:05.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e4c0c6340 0x7f2e4c14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e4c0c7c00 0x7f2e4c14b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c0c9a80 con 0x7f2e4c151ad0 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c072db0 con 0x7f2e4c0c7c00 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e52e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4c0c9190 con 0x7f2e4c0c6340 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e50e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e50e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48140/0 (socket says 172.21.15.92:48140) 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e50e00640 1 -- 172.21.15.92:0/586782405 learned_addr learned my addr 172.21.15.92:0/586782405 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e4b400640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e4c0c7c00 0x7f2e4c14b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.419+0000 7f2e4be00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e4c0c6340 0x7f2e4c14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 -- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e4c0c6340 msgr2=0x7f2e4c14d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e4c0c6340 0x7f2e4c14d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 -- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e4c0c7c00 msgr2=0x7f2e4c14b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e4c0c7c00 0x7f2e4c14b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 -- 172.21.15.92:0/586782405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4c1ef280 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e50e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e4402bf70 tx=0x7f2e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44034040 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4403aa20 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4c14a0d0 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44006e10 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.420+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4c1eefa0 con 0x7f2e4c151ad0 2024-03-19T02:55:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.421+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e4c0768a0 con 0x7f2e4c151ad0 2024-03-19T02:55:05.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.422+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e44037070 con 0x7f2e4c151ad0 2024-03-19T02:55:05.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.423+0000 7f2e43e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 0x7f2e2806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.423+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2e440ac6c0 con 0x7f2e4c151ad0 2024-03-19T02:55:05.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.423+0000 7f2e4be00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 0x7f2e2806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.424+0000 7f2e4be00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 0x7f2e2806e0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c000f60 tx=0x7f2e3c007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.428+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e4403f070 con 0x7f2e4c151ad0 2024-03-19T02:55:05.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.605+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e4c1eed40 con 0x7f2e2806bc00 2024-03-19T02:55:05.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.612+0000 7f2e43e00640 1 -- 172.21.15.92:0/586782405 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252024 (secure 0 0 0) 0x7f2e3c0014f0 con 0x7f2e2806bc00 2024-03-19T02:55:05.616 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:05.617 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:05.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 msgr2=0x7f2e2806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 0x7f2e2806e0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c000f60 tx=0x7f2e3c007d50 comp rx=0 tx=0).stop 2024-03-19T02:55:05.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 msgr2=0x7f2e4c1544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e4402bf70 tx=0x7f2e440046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 shutdown_connections 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2e2806bc00 0x7f2e2806e0c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e4c151ad0 0x7f2e4c1544c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e4c0c7c00 0x7f2e4c14b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 --2- 172.21.15.92:0/586782405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e4c0c6340 0x7f2e4c14d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 >> 172.21.15.92:0/586782405 conn(0x7f2e4c0cd200 msgr2=0x7f2e4c151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 shutdown_connections 2024-03-19T02:55:05.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.617+0000 7f2e52e00640 1 -- 172.21.15.92:0/586782405 wait complete. 2024-03-19T02:55:05.635 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-19T02:55:03.837574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217920,"kb_used_data":1792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756800,"statfs":{"total":773094113280,"available":772870963200,"internally_reserved":0,"allocated":1835008,"data_stored":1753028,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.663089"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380801+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380801+0000","last_peered":"2024-03-19T02:54:57.380801+0000","last_clean":"2024-03-19T02:54:57.380801+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380801+0000","last_undegraded":"2024-03-19T02:54:57.380801+0000","last_fullsized":"2024-03-19T02:54:57.380801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081567+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081567+0000","last_peered":"2024-03-19T02:54:59.081567+0000","last_clean":"2024-03-19T02:54:59.081567+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081567+0000","last_undegraded":"2024-03-19T02:54:59.081567+0000","last_fullsized":"2024-03-19T02:54:59.081567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.182131+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:55:01.182131+0000","last_peered":"2024-03-19T02:55:01.182131+0000","last_clean":"2024-03-19T02:55:01.182131+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:55:01.182131+0000","last_undegraded":"2024-03-19T02:55:01.182131+0000","last_fullsized":"2024-03-19T02:55:01.182131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183040+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:55:01.183040+0000","last_peered":"2024-03-19T02:55:01.183040+0000","last_clean":"2024-03-19T02:55:01.183040+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:55:01.183040+0000","last_undegraded":"2024-03-19T02:55:01.183040+0000","last_fullsized":"2024-03-19T02:55:01.183040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183309+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:55:01.183309+0000","last_peered":"2024-03-19T02:55:01.183309+0000","last_clean":"2024-03-19T02:55:01.183309+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:55:01.183309+0000","last_undegraded":"2024-03-19T02:55:01.183309+0000","last_fullsized":"2024-03-19T02:55:01.183309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.186421+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:55:01.186421+0000","last_peered":"2024-03-19T02:55:01.186421+0000","last_clean":"2024-03-19T02:55:01.186421+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:55:01.186421+0000","last_undegraded":"2024-03-19T02:55:01.186421+0000","last_fullsized":"2024-03-19T02:55:01.186421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197959+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197959+0000","last_peered":"2024-03-19T02:54:57.197959+0000","last_clean":"2024-03-19T02:54:57.197959+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197959+0000","last_undegraded":"2024-03-19T02:54:57.197959+0000","last_fullsized":"2024-03-19T02:54:57.197959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188015+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:55:01.188015+0000","last_peered":"2024-03-19T02:55:01.188015+0000","last_clean":"2024-03-19T02:55:01.188015+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:55:01.188015+0000","last_undegraded":"2024-03-19T02:55:01.188015+0000","last_fullsized":"2024-03-19T02:55:01.188015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201693+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201693+0000","last_peered":"2024-03-19T02:54:57.201693+0000","last_clean":"2024-03-19T02:54:57.201693+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201693+0000","last_undegraded":"2024-03-19T02:54:57.201693+0000","last_fullsized":"2024-03-19T02:54:57.201693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201716+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201716+0000","last_peered":"2024-03-19T02:54:57.201716+0000","last_clean":"2024-03-19T02:54:57.201716+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201716+0000","last_undegraded":"2024-03-19T02:54:57.201716+0000","last_fullsized":"2024-03-19T02:54:57.201716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177520+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:55:01.177520+0000","last_peered":"2024-03-19T02:55:01.177520+0000","last_clean":"2024-03-19T02:55:01.177520+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:55:01.177520+0000","last_undegraded":"2024-03-19T02:55:01.177520+0000","last_fullsized":"2024-03-19T02:55:01.177520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177763+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:55:01.177763+0000","last_peered":"2024-03-19T02:55:01.177763+0000","last_clean":"2024-03-19T02:55:01.177763+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:55:01.177763+0000","last_undegraded":"2024-03-19T02:55:01.177763+0000","last_fullsized":"2024-03-19T02:55:01.177763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177777+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:55:01.177777+0000","last_peered":"2024-03-19T02:55:01.177777+0000","last_clean":"2024-03-19T02:55:01.177777+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:55:01.177777+0000","last_undegraded":"2024-03-19T02:55:01.177777+0000","last_fullsized":"2024-03-19T02:55:01.177777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.178273+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:55:01.178273+0000","last_peered":"2024-03-19T02:55:01.178273+0000","last_clean":"2024-03-19T02:55:01.178273+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:55:01.178273+0000","last_undegraded":"2024-03-19T02:55:01.178273+0000","last_fullsized":"2024-03-19T02:55:01.178273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381552+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381552+0000","last_peered":"2024-03-19T02:54:57.381552+0000","last_clean":"2024-03-19T02:54:57.381552+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381552+0000","last_undegraded":"2024-03-19T02:54:57.381552+0000","last_fullsized":"2024-03-19T02:54:57.381552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188465+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:55:01.188465+0000","last_peered":"2024-03-19T02:55:01.188465+0000","last_clean":"2024-03-19T02:55:01.188465+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:55:01.188465+0000","last_undegraded":"2024-03-19T02:55:01.188465+0000","last_fullsized":"2024-03-19T02:55:01.188465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661884,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":661940,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:05.639 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T02:55:05.639 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T02:55:05.639 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-19T02:55:05.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-19T02:55:05.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.878+0000 7f4fc3200640 1 Processor -- start 2024-03-19T02:55:05.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.878+0000 7f4fc3200640 1 -- start start 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4fbc06b240 0x7f4fbc0c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4fbc0c5490 0x7f4fbc06b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4fbc158920 con 0x7f4fbc0c5490 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f4fbc0c58d0 con 0x7f4fbc06b240 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.879+0000 7f4fc3200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc158360 con 0x7f4fbc0c38b0 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4fbc0c5490 0x7f4fbc06b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4fbc0c5490 0x7f4fbc06b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42360/0 (socket says 172.21.15.92:42360) 2024-03-19T02:55:05.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 -- 172.21.15.92:0/1560241926 learned_addr learned my addr 172.21.15.92:0/1560241926 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:05.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fbbe00640 1 --2- 172.21.15.92:0/1560241926 >> v2:172.21.15.159:3300/0 conn(0x7f4fbc06b240 0x7f4fbc0c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fc1200640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 -- 172.21.15.92:0/1560241926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 msgr2=0x7f4fbc1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:05.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 -- 172.21.15.92:0/1560241926 >> v2:172.21.15.159:3300/0 conn(0x7f4fbc06b240 msgr2=0x7f4fbc0c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:05.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1560241926 >> v2:172.21.15.159:3300/0 conn(0x7f4fbc06b240 0x7f4fbc0c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fb3400640 1 -- 172.21.15.92:0/1560241926 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc158640 con 0x7f4fbc0c5490 2024-03-19T02:55:05.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.880+0000 7f4fc1200640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:05.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.881+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc06b620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0ccf60 tx=0x7f4fa402e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2ade27f213112f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.882+0000 7f4fbb400640 1 -- 172.21.15.92:0/1560241926 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa403b070 con 0x7f4fbc0c5490 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.882+0000 7f4fbb400640 1 -- 172.21.15.92:0/1560241926 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4036440 con 0x7f4fbc0c5490 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.882+0000 7f4fbb400640 1 -- 172.21.15.92:0/1560241926 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4036810 con 0x7f4fbc0c5490 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.882+0000 7f4fc3200640 1 -- 172.21.15.92:0/1560241926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc0c5490 msgr2=0x7f4fbc06b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.882+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc06b620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0ccf60 tx=0x7f4fa402e0d0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 -- 172.21.15.92:0/1560241926 shutdown_connections 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1560241926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc06b620 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1560241926 >> v2:172.21.15.159:3300/0 conn(0x7f4fbc06b240 0x7f4fbc0c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 -- 172.21.15.92:0/1560241926 >> 172.21.15.92:0/1560241926 conn(0x7f4fbc0cc1f0 msgr2=0x7f4fbc0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 -- 172.21.15.92:0/1560241926 shutdown_connections 2024-03-19T02:55:05.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.883+0000 7f4fc3200640 1 -- 172.21.15.92:0/1560241926 wait complete. 2024-03-19T02:55:05.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.884+0000 7f4fc3200640 1 Processor -- start 2024-03-19T02:55:05.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.884+0000 7f4fc3200640 1 -- start start 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc1f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc0c71e0 con 0x7f4fbc06b240 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc0c6f00 con 0x7f4fbc0c5490 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fc3200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc151fb0 con 0x7f4fbc0c38b0 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fbbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fbbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42376/0 (socket says 172.21.15.92:42376) 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.885+0000 7f4fbbe00640 1 -- 172.21.15.92:0/1425493768 learned_addr learned my addr 172.21.15.92:0/1425493768 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fc1200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc1f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 -- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 msgr2=0x7f4fbc1f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 -- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fbc0c5490 msgr2=0x7f4fbc1f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc1f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 -- 172.21.15.92:0/1425493768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc15b430 con 0x7f4fbc06b240 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fbbe00640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00ef00 tx=0x7f4fac00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac018020 con 0x7f4fbc06b240 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fac014400 con 0x7f4fbc06b240 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0147d0 con 0x7f4fbc06b240 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fbc159520 con 0x7f4fbc06b240 2024-03-19T02:55:05.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.886+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fbc159ae0 con 0x7f4fbc06b240 2024-03-19T02:55:05.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.887+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fbc0684c0 con 0x7f4fbc06b240 2024-03-19T02:55:05.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.888+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4fac005d10 con 0x7f4fbc06b240 2024-03-19T02:55:05.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.889+0000 7f4fb8c00640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 0x7f4f9806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:05.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.889+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4fac08b100 con 0x7f4fbc06b240 2024-03-19T02:55:05.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.889+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 0x7f4f9806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:05.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.890+0000 7f4fb3400640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 0x7f4f9806e0c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4038000 tx=0x7f4fa4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:05.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:05.893+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4fac022070 con 0x7f4fbc06b240 2024-03-19T02:55:06.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.111+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4fbc0c7800 con 0x7f4fbc06b240 2024-03-19T02:55:06.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.111+0000 7f4fb8c00640 1 -- 172.21.15.92:0/1425493768 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4fac059b50 con 0x7f4fbc06b240 2024-03-19T02:55:06.113 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:06.113 INFO:teuthology.orchestra.run.smithi092.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.113+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 msgr2=0x7f4f9806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.113+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 0x7f4f9806e0c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4038000 tx=0x7f4fa4008040 comp rx=0 tx=0).stop 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.113+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 msgr2=0x7f4fbc1e79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.113+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00ef00 tx=0x7f4fac00b480 comp rx=0 tx=0).stop 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 shutdown_connections 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4f9806bc00 0x7f4f9806e0c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fbc0c5490 0x7f4fbc1f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fbc0c38b0 0x7f4fbc1f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 --2- 172.21.15.92:0/1425493768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4fbc06b240 0x7f4fbc1e79d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.116 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 >> 172.21.15.92:0/1425493768 conn(0x7f4fbc0cc1f0 msgr2=0x7f4fbc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:06.116 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 shutdown_connections 2024-03-19T02:55:06.116 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.114+0000 7f4fc3200640 1 -- 172.21.15.92:0/1425493768 wait complete. 2024-03-19T02:55:06.133 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-19T02:55:06.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:06.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.340+0000 7f6f03600640 1 Processor -- start 2024-03-19T02:55:06.342 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.340+0000 7f6f03600640 1 -- start start 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f03600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6efc152bf0 0x7f6efc06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f03600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc0c6470 0x7f6efc0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f03600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6efc0c9a80 con 0x7f6efc1593d0 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f03600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6efc0c6c90 con 0x7f6efc152bf0 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f03600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc159910 con 0x7f6efc0c6470 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f00c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6efc152bf0 0x7f6efc06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f00c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6efc152bf0 0x7f6efc06eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42378/0 (socket says 172.21.15.92:42378) 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f00c00640 1 -- 172.21.15.92:0/232085753 learned_addr learned my addr 172.21.15.92:0/232085753 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6efbe00640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc0c6470 0x7f6efc0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f00c00640 1 -- 172.21.15.92:0/232085753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc0c6470 msgr2=0x7f6efc0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.342+0000 7f6f00c00640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc0c6470 0x7f6efc0c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.343+0000 7f6f00c00640 1 -- 172.21.15.92:0/232085753 >> v1:172.21.15.92:6789/0 conn(0x7f6efc1593d0 legacy=0x7f6efc0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.343+0000 7f6f00c00640 1 -- 172.21.15.92:0/232085753 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc0c8650 con 0x7f6efc152bf0 2024-03-19T02:55:06.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.344+0000 7f6f00c00640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc152bf0 0x7f6efc06eef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009870 tx=0x7f6ef002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261762d7fa57c4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.344+0000 7f6efb400640 1 -- 172.21.15.92:0/232085753 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0035020 con 0x7f6efc152bf0 2024-03-19T02:55:06.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.344+0000 7f6efb400640 1 -- 172.21.15.92:0/232085753 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef0031640 con 0x7f6efc152bf0 2024-03-19T02:55:06.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.344+0000 7f6efb400640 1 -- 172.21.15.92:0/232085753 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef002d040 con 0x7f6efc152bf0 2024-03-19T02:55:06.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 -- 172.21.15.92:0/232085753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc152bf0 msgr2=0x7f6efc06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc152bf0 0x7f6efc06eef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009870 tx=0x7f6ef002b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 -- 172.21.15.92:0/232085753 shutdown_connections 2024-03-19T02:55:06.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc0c6470 0x7f6efc0c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 --2- 172.21.15.92:0/232085753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc152bf0 0x7f6efc06eef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 -- 172.21.15.92:0/232085753 >> 172.21.15.92:0/232085753 conn(0x7f6efc0c8d90 msgr2=0x7f6efc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:06.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 -- 172.21.15.92:0/232085753 shutdown_connections 2024-03-19T02:55:06.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.345+0000 7f6f03600640 1 -- 172.21.15.92:0/232085753 wait complete. 2024-03-19T02:55:06.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.347+0000 7f6f03600640 1 Processor -- start 2024-03-19T02:55:06.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.347+0000 7f6f03600640 1 -- start start 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6efc152bf0 0x7f6efc0c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc1593d0 0x7f6efc14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc06ea60 con 0x7f6efc0c6470 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc06e4a0 con 0x7f6efc152bf0 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f03600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc06e6f0 con 0x7f6efc1593d0 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f00c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f00c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42386/0 (socket says 172.21.15.92:42386) 2024-03-19T02:55:06.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f00c00640 1 -- 172.21.15.92:0/3519317614 learned_addr learned my addr 172.21.15.92:0/3519317614 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:06.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6efbe00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6efc152bf0 0x7f6efc0c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.348+0000 7f6f01600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc1593d0 0x7f6efc14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 -- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc1593d0 msgr2=0x7f6efc14ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc1593d0 0x7f6efc14ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 -- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6efc152bf0 msgr2=0x7f6efc0c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6efc152bf0 0x7f6efc0c27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 -- 172.21.15.92:0/3519317614 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc14daa0 con 0x7f6efc0c6470 2024-03-19T02:55:06.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f00c00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0000c00 tx=0x7f6ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0040070 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef00399f0 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0035040 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6efc147490 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.349+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6efc072bb0 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.350+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6efc0684c0 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.351+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ef004b060 con 0x7f6efc0c6470 2024-03-19T02:55:06.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.352+0000 7f6ef8c00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 0x7f6edc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.352+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ef00ac240 con 0x7f6efc0c6470 2024-03-19T02:55:06.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.352+0000 7f6efbe00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 0x7f6edc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.353+0000 7f6efbe00640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 0x7f6edc06e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6efc158970 tx=0x7f6ee8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.356+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6efc14daa0 con 0x7f6efc0c6470 2024-03-19T02:55:06.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.542+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6efc06f950 con 0x7f6efc0c6470 2024-03-19T02:55:06.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.545+0000 7f6ef8c00640 1 -- 172.21.15.92:0/3519317614 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f6ef007ad20 con 0x7f6efc0c6470 2024-03-19T02:55:06.547 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:06.547 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:06.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 msgr2=0x7f6edc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 0x7f6edc06e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6efc158970 tx=0x7f6ee8009190 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 msgr2=0x7f6efc0cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0000c00 tx=0x7f6ef00046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 shutdown_connections 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6edc06bbd0 0x7f6edc06e090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6efc1593d0 0x7f6efc14ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6efc152bf0 0x7f6efc0c27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 --2- 172.21.15.92:0/3519317614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6efc0c6470 0x7f6efc0cbb00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 >> 172.21.15.92:0/3519317614 conn(0x7f6efc0c8d90 msgr2=0x7f6efc151240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 shutdown_connections 2024-03-19T02:55:06.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.548+0000 7f6f03600640 1 -- 172.21.15.92:0/3519317614 wait complete. 2024-03-19T02:55:06.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:55:06.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:55:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.772+0000 7f74dac00640 1 Processor -- start 2024-03-19T02:55:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.773+0000 7f74dac00640 1 -- start start 2024-03-19T02:55:06.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74dac00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f74d4152a80 0x7f74d40c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74dac00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d40c6d30 con 0x7f74d41526a0 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74dac00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d40c6ff0 con 0x7f74d40c6460 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74dac00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f74d40c72d0 con 0x7f74d4152a80 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74d9800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f74d41526a0 0x7f74d4158800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46346/0 (socket says 172.21.15.92:46346) 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74d9800640 1 -- 172.21.15.92:0/2043268441 learned_addr learned my addr 172.21.15.92:0/2043268441 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.774+0000 7f74d8e00640 1 --2- 172.21.15.92:0/2043268441 >> v2:172.21.15.159:3300/0 conn(0x7f74d4152a80 0x7f74d40c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1335714661 0 0) 0x7f74d40c6ff0 con 0x7f74d40c6460 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74bc003520 con 0x7f74d40c6460 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d8e00640 1 -- 172.21.15.92:0/2043268441 >> v1:172.21.15.159:6789/0 conn(0x7f74d40c6460 legacy=0x7f74d406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d8e00640 1 -- 172.21.15.92:0/2043268441 >> v1:172.21.15.92:6789/0 conn(0x7f74d41526a0 legacy=0x7f74d4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177822270 0 0) 0x7f74d40c6d30 con 0x7f74d41526a0 2024-03-19T02:55:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.775+0000 7f74d8e00640 1 -- 172.21.15.92:0/2043268441 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d40c43d0 con 0x7f74d4152a80 2024-03-19T02:55:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.776+0000 7f74d8e00640 1 --2- 172.21.15.92:0/2043268441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d4152a80 0x7f74d40c3790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74c00097a0 tx=0x7f74c002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95b15ebe88c343a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.776+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0034070 con 0x7f74d4152a80 2024-03-19T02:55:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.776+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c0030610 con 0x7f74d4152a80 2024-03-19T02:55:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.776+0000 7f74d3e00640 1 -- 172.21.15.92:0/2043268441 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c00309c0 con 0x7f74d4152a80 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 -- 172.21.15.92:0/2043268441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d4152a80 msgr2=0x7f74d40c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 --2- 172.21.15.92:0/2043268441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d4152a80 0x7f74d40c3790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74c00097a0 tx=0x7f74c002b960 comp rx=0 tx=0).stop 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 -- 172.21.15.92:0/2043268441 shutdown_connections 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 --2- 172.21.15.92:0/2043268441 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d4152a80 0x7f74d40c3790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 -- 172.21.15.92:0/2043268441 >> 172.21.15.92:0/2043268441 conn(0x7f74d406bac0 msgr2=0x7f74d40cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 -- 172.21.15.92:0/2043268441 shutdown_connections 2024-03-19T02:55:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.777+0000 7f74dac00640 1 -- 172.21.15.92:0/2043268441 wait complete. 2024-03-19T02:55:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.779+0000 7f74dac00640 1 Processor -- start 2024-03-19T02:55:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.779+0000 7f74dac00640 1 -- start start 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d40c6460 0x7f74d40cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74d41526a0 0x7f74d4155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74d40c3cd0 con 0x7f74d4152a80 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74d40c4080 con 0x7f74d40c6460 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74dac00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74d40c8a50 con 0x7f74d41526a0 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42394/0 (socket says 172.21.15.92:42394) 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 -- 172.21.15.92:0/2537332667 learned_addr learned my addr 172.21.15.92:0/2537332667 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74d9800640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d40c6460 0x7f74d40cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74d8e00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74d41526a0 0x7f74d4155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 -- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74d41526a0 msgr2=0x7f74d4155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74d41526a0 0x7f74d4155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 -- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d40c6460 msgr2=0x7f74d40cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d40c6460 0x7f74d40cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.780+0000 7f74da200640 1 -- 172.21.15.92:0/2537332667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d41ee640 con 0x7f74d4152a80 2024-03-19T02:55:06.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74da200640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c7c0 tx=0x7f74cc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc003b70 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc003d10 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d41565d0 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00aa50 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.781+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d4156b30 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.782+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d404f580 con 0x7f74d4152a80 2024-03-19T02:55:06.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.783+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f74cc00abf0 con 0x7f74d4152a80 2024-03-19T02:55:06.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.784+0000 7f74d1600640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 0x7f74b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:06.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.784+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f74cc08a950 con 0x7f74d4152a80 2024-03-19T02:55:06.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.784+0000 7f74d9800640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 0x7f74b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:06.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.785+0000 7f74d9800640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 0x7f74b406e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f74c8000f60 tx=0x7f74c8007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.788+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74d41565d0 con 0x7f74d4152a80 2024-03-19T02:55:06.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.985+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74d40c75e0 con 0x7f74d4152a80 2024-03-19T02:55:06.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.985+0000 7f74d1600640 1 -- 172.21.15.92:0/2537332667 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f74cc058df0 con 0x7f74d4152a80 2024-03-19T02:55:06.987 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:55:06.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 msgr2=0x7f74b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 0x7f74b406e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f74c8000f60 tx=0x7f74c8007d50 comp rx=0 tx=0).stop 2024-03-19T02:55:06.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 msgr2=0x7f74d4157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c7c0 tx=0x7f74cc00cc80 comp rx=0 tx=0).stop 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 shutdown_connections 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f74b406bbd0 0x7f74b406e090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74d4152a80 0x7f74d4157610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74d41526a0 0x7f74d4155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 --2- 172.21.15.92:0/2537332667 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74d40c6460 0x7f74d40cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 >> 172.21.15.92:0/2537332667 conn(0x7f74d406bac0 msgr2=0x7f74d4150000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 shutdown_connections 2024-03-19T02:55:06.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:06.988+0000 7f74dac00640 1 -- 172.21.15.92:0/2537332667 wait complete. 2024-03-19T02:55:07.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:55:07.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.214+0000 7f9bca800640 1 Processor -- start 2024-03-19T02:55:07.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.214+0000 7f9bca800640 1 -- start start 2024-03-19T02:55:07.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.215+0000 7f9bca800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9bc4154b10 0x7f9bc40cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.215+0000 7f9bca800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 0x7f9bc415c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.215+0000 7f9bca800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc415c550 con 0x7f9bc40c63e0 2024-03-19T02:55:07.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.215+0000 7f9bca800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f9bc415b9e0 con 0x7f9bc4154b10 2024-03-19T02:55:07.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.215+0000 7f9bca800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9bc40c5340 con 0x7f9bc415b550 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9bc4154b10 0x7f9bc40cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc2a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 0x7f9bc415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9bc4154b10 0x7f9bc40cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36628/0 (socket says 172.21.15.92:36628) 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc2a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 0x7f9bc415c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49666/0 (socket says 172.21.15.92:49666) 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 -- 172.21.15.92:0/358425216 learned_addr learned my addr 172.21.15.92:0/358425216 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:07.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3e00640 1 --1- 172.21.15.92:0/358425216 >> v1:172.21.15.159:6789/0 conn(0x7f9bc40c63e0 0x7f9bc406c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52380/0 (socket says 172.21.15.92:52380) 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 -- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 msgr2=0x7f9bc415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 --2- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 0x7f9bc415c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 -- 172.21.15.92:0/358425216 >> v1:172.21.15.159:6789/0 conn(0x7f9bc40c63e0 legacy=0x7f9bc406c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc3400640 1 -- 172.21.15.92:0/358425216 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc40c5620 con 0x7f9bc4154b10 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.216+0000 7f9bc2000640 1 -- 172.21.15.92:0/358425216 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353385747 0 0) 0x7f9bc415c550 con 0x7f9bc40c63e0 2024-03-19T02:55:07.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.218+0000 7f9bc3400640 1 --2- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc4154b10 0x7f9bc40cae40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009e50 tx=0x7f9bb402bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e846b73854365e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.218+0000 7f9bc2000640 1 -- 172.21.15.92:0/358425216 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb402cb90 con 0x7f9bc4154b10 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.218+0000 7f9bc2000640 1 -- 172.21.15.92:0/358425216 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bb402cd30 con 0x7f9bc4154b10 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.218+0000 7f9bc2000640 1 -- 172.21.15.92:0/358425216 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4034610 con 0x7f9bc4154b10 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 -- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc4154b10 msgr2=0x7f9bc40cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 --2- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc4154b10 0x7f9bc40cae40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009e50 tx=0x7f9bb402bff0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 -- 172.21.15.92:0/358425216 shutdown_connections 2024-03-19T02:55:07.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 --2- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc415b550 0x7f9bc415c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 --2- 172.21.15.92:0/358425216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc4154b10 0x7f9bc40cae40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 -- 172.21.15.92:0/358425216 >> 172.21.15.92:0/358425216 conn(0x7f9bc4072970 msgr2=0x7f9bc4066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:07.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 -- 172.21.15.92:0/358425216 shutdown_connections 2024-03-19T02:55:07.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.219+0000 7f9bca800640 1 -- 172.21.15.92:0/358425216 wait complete. 2024-03-19T02:55:07.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.220+0000 7f9bca800640 1 Processor -- start 2024-03-19T02:55:07.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 -- start start 2024-03-19T02:55:07.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 0x7f9bc40c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 0x7f9bc40b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc40c5ec0 con 0x7f9bc4154b10 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc40c5be0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.221+0000 7f9bca800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9bc4067ca0 con 0x7f9bc40c63e0 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 0x7f9bc40c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42400/0 (socket says 172.21.15.92:42400) 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 0x7f9bc40b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc2a00640 1 -- 172.21.15.92:0/2186259642 learned_addr learned my addr 172.21.15.92:0/2186259642 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 -- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 msgr2=0x7f9bc40c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 0x7f9bc40c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 -- 172.21.15.92:0/2186259642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 msgr2=0x7f9bc4150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 -- 172.21.15.92:0/2186259642 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc40cbd70 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3400640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 0x7f9bc40c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:07.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc2a00640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:07.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.222+0000 7f9bc3e00640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 0x7f9bc40b4f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c450 tx=0x7f9bbc00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.223+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0074e0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.223+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bbc0047e0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.223+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc40cc050 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.223+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc004bd0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.223+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc40cc610 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.224+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc40684c0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.225+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9bbc011070 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.226+0000 7f9bbb400640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 0x7f9ba406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.226+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9bbc08a4a0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.226+0000 7f9bc3400640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 0x7f9ba406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.227+0000 7f9bc3400640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 0x7f9ba406e0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4035000 tx=0x7f9bb4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.230+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9bbc007e20 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.426+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9bc40cb420 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.426+0000 7f9bbb400640 1 -- 172.21.15.92:0/2186259642 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f9bbc0588a0 con 0x7f9bc40b4ae0 2024-03-19T02:55:07.428 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T02:55:07.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 msgr2=0x7f9ba406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 0x7f9ba406e0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4035000 tx=0x7f9bb4030000 comp rx=0 tx=0).stop 2024-03-19T02:55:07.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 msgr2=0x7f9bc40b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 0x7f9bc40b4f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c450 tx=0x7f9bbc00c910 comp rx=0 tx=0).stop 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 shutdown_connections 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ba406bc00 0x7f9ba406e0c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9bc40b4ae0 0x7f9bc40b4f60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4154b10 0x7f9bc4150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 --2- 172.21.15.92:0/2186259642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9bc40c63e0 0x7f9bc40c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 >> 172.21.15.92:0/2186259642 conn(0x7f9bc4072970 msgr2=0x7f9bc4153250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 shutdown_connections 2024-03-19T02:55:07.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.429+0000 7f9bca800640 1 -- 172.21.15.92:0/2186259642 wait complete. 2024-03-19T02:55:07.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:55:07.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.663+0000 7f2f53c00640 1 Processor -- start 2024-03-19T02:55:07.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.663+0000 7f2f53c00640 1 -- start start 2024-03-19T02:55:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.664+0000 7f2f53c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2f4c152bf0 0x7f2f4c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f53c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2f4c1593d0 0x7f2f4c0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f53c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f4c0c9a80 con 0x7f2f4c0c6470 2024-03-19T02:55:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f53c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4c0c6c90 con 0x7f2f4c152bf0 2024-03-19T02:55:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f53c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4c159910 con 0x7f2f4c1593d0 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2f4c152bf0 0x7f2f4c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2f4c1593d0 0x7f2f4c0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f4be00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f2f4c0c6470 0x7f2f4c0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52390/0 (socket says 172.21.15.92:52390) 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f4be00640 1 -- 172.21.15.92:0/1723233847 learned_addr learned my addr 172.21.15.92:0/1723233847 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51c00640 1 -- 172.21.15.92:0/1723233847 >> v2:172.21.15.92:3300/0 conn(0x7f2f4c152bf0 msgr2=0x7f2f4c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51c00640 1 --2- 172.21.15.92:0/1723233847 >> v2:172.21.15.92:3300/0 conn(0x7f2f4c152bf0 0x7f2f4c06eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51c00640 1 -- 172.21.15.92:0/1723233847 >> v1:172.21.15.159:6789/0 conn(0x7f2f4c0c6470 legacy=0x7f2f4c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.665+0000 7f2f51c00640 1 -- 172.21.15.92:0/1723233847 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c0c8650 con 0x7f2f4c1593d0 2024-03-19T02:55:07.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.666+0000 7f2f51c00640 1 --2- 172.21.15.92:0/1723233847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c1593d0 0x7f2f4c0c4690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0c5070 tx=0x7f2f4402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2dee8f2a5a621da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f4b400640 1 -- 172.21.15.92:0/1723233847 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44039070 con 0x7f2f4c1593d0 2024-03-19T02:55:07.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f4b400640 1 -- 172.21.15.92:0/1723233847 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4402cbf0 con 0x7f2f4c1593d0 2024-03-19T02:55:07.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f4b400640 1 -- 172.21.15.92:0/1723233847 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44034440 con 0x7f2f4c1593d0 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 -- 172.21.15.92:0/1723233847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c1593d0 msgr2=0x7f2f4c0c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 --2- 172.21.15.92:0/1723233847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c1593d0 0x7f2f4c0c4690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0c5070 tx=0x7f2f4402be40 comp rx=0 tx=0).stop 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 -- 172.21.15.92:0/1723233847 shutdown_connections 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 --2- 172.21.15.92:0/1723233847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c1593d0 0x7f2f4c0c4690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 --2- 172.21.15.92:0/1723233847 >> v2:172.21.15.92:3300/0 conn(0x7f2f4c152bf0 0x7f2f4c06eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.667+0000 7f2f53c00640 1 -- 172.21.15.92:0/1723233847 >> 172.21.15.92:0/1723233847 conn(0x7f2f4c0c8d90 msgr2=0x7f2f4c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.668+0000 7f2f53c00640 1 -- 172.21.15.92:0/1723233847 shutdown_connections 2024-03-19T02:55:07.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.668+0000 7f2f53c00640 1 -- 172.21.15.92:0/1723233847 wait complete. 2024-03-19T02:55:07.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.670+0000 7f2f53c00640 1 Processor -- start 2024-03-19T02:55:07.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.670+0000 7f2f53c00640 1 -- start start 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 0x7f2f4c14e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 0x7f2f4c147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c06ea60 con 0x7f2f4c0c6470 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c06e4a0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f53c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4c06e6f0 con 0x7f2f4c1593d0 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f4be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f51200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 0x7f2f4c14e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f51c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 0x7f2f4c147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f51200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 0x7f2f4c14e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42412/0 (socket says 172.21.15.92:42412) 2024-03-19T02:55:07.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f51c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 0x7f2f4c147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49688/0 (socket says 172.21.15.92:49688) 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f4be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36668/0 (socket says 172.21.15.92:36668) 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.671+0000 7f2f51200640 1 -- 172.21.15.92:0/3535165383 learned_addr learned my addr 172.21.15.92:0/3535165383 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 -- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 msgr2=0x7f2f4c147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 0x7f2f4c147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 -- 172.21.15.92:0/3535165383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 msgr2=0x7f2f4c14e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 0x7f2f4c14e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 -- 172.21.15.92:0/3535165383 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c14ce90 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f4be00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2f40000f60 tx=0x7f2f4000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40017070 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f40005ce0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40007310 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.672+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4c1f2ae0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.673+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4c14d410 con 0x7f2f4c152bf0 2024-03-19T02:55:07.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.674+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4c14d650 con 0x7f2f4c152bf0 2024-03-19T02:55:07.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.674+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2f4001a070 con 0x7f2f4c152bf0 2024-03-19T02:55:07.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.675+0000 7f2f48c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 0x7f2f3006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:07.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.675+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2f40089de0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.675+0000 7f2f51200640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 0x7f2f3006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:07.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.676+0000 7f2f51200640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 0x7f2f3006e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009c00 tx=0x7f2f3c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:07.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.681+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2f4c14d650 con 0x7f2f4c152bf0 2024-03-19T02:55:07.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.876+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2f4c0684c0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.877+0000 7f2f48c00640 1 -- 172.21.15.92:0/3535165383 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f2f400581f0 con 0x7f2f4c152bf0 2024-03-19T02:55:07.879 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T02:55:07.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 msgr2=0x7f2f3006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 0x7f2f3006e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009c00 tx=0x7f2f3c009190 comp rx=0 tx=0).stop 2024-03-19T02:55:07.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 msgr2=0x7f2f4c1f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2f40000f60 tx=0x7f2f4000c800 comp rx=0 tx=0).stop 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 shutdown_connections 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f3006bc00 0x7f2f3006e0c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f4c1593d0 0x7f2f4c147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f4c152bf0 0x7f2f4c1f25a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 --2- 172.21.15.92:0/3535165383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f4c0c6470 0x7f2f4c14e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 >> 172.21.15.92:0/3535165383 conn(0x7f2f4c0c8d90 msgr2=0x7f2f4c151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 shutdown_connections 2024-03-19T02:55:07.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:07.880+0000 7f2f53c00640 1 -- 172.21.15.92:0/3535165383 wait complete. 2024-03-19T02:55:07.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:08.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.099+0000 7f8067200640 1 Processor -- start 2024-03-19T02:55:08.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.099+0000 7f8067200640 1 -- start start 2024-03-19T02:55:08.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8067200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f80680cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8067200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f80680cd1c0 0x7f80680cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8067200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80680b3290 con 0x7f80680b34c0 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8067200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f80680cccc0 con 0x7f80680cd1c0 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8067200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f80680b2cf0 con 0x7f8068154de0 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8066800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f80680cd1c0 0x7f80680cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8066800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f80680cd1c0 0x7f80680cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42424/0 (socket says 172.21.15.92:42424) 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8065e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f80680cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.101+0000 7f8066800640 1 -- 172.21.15.92:0/1448980272 learned_addr learned my addr 172.21.15.92:0/1448980272 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:08.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.102+0000 7f8066800640 1 -- 172.21.15.92:0/1448980272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 msgr2=0x7f80680cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:08.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.102+0000 7f8066800640 1 --2- 172.21.15.92:0/1448980272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f80680cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.102+0000 7f8066800640 1 -- 172.21.15.92:0/1448980272 >> v1:172.21.15.92:6789/0 conn(0x7f80680b34c0 legacy=0x7f806806bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.102+0000 7f8066800640 1 -- 172.21.15.92:0/1448980272 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806815ef90 con 0x7f80680cd1c0 2024-03-19T02:55:08.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.103+0000 7f8066800640 1 --2- 172.21.15.92:0/1448980272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f80680cd650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8060009e50 tx=0x7f806002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb1acde56d284f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.103+0000 7f8065400640 1 -- 172.21.15.92:0/1448980272 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806002cb00 con 0x7f80680cd1c0 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.103+0000 7f8065400640 1 -- 172.21.15.92:0/1448980272 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f806002cca0 con 0x7f80680cd1c0 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.103+0000 7f8065400640 1 -- 172.21.15.92:0/1448980272 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060034610 con 0x7f80680cd1c0 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 -- 172.21.15.92:0/1448980272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 msgr2=0x7f80680cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 --2- 172.21.15.92:0/1448980272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f80680cd650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8060009e50 tx=0x7f806002bf60 comp rx=0 tx=0).stop 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 -- 172.21.15.92:0/1448980272 shutdown_connections 2024-03-19T02:55:08.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 --2- 172.21.15.92:0/1448980272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f80680cd650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 --2- 172.21.15.92:0/1448980272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f80680cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 -- 172.21.15.92:0/1448980272 >> 172.21.15.92:0/1448980272 conn(0x7f8068067130 msgr2=0x7f8068067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:08.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 -- 172.21.15.92:0/1448980272 shutdown_connections 2024-03-19T02:55:08.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.104+0000 7f8067200640 1 -- 172.21.15.92:0/1448980272 wait complete. 2024-03-19T02:55:08.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.106+0000 7f8067200640 1 Processor -- start 2024-03-19T02:55:08.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.106+0000 7f8067200640 1 -- start start 2024-03-19T02:55:08.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.106+0000 7f8067200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8067200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f806814e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8067200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f8068151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8067200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f806815e4b0 con 0x7f80680cd1c0 2024-03-19T02:55:08.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8067200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f806815e1d0 con 0x7f80680b34c0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8067200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f806815e790 con 0x7f8068154de0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36680/0 (socket says 172.21.15.92:36680) 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 -- 172.21.15.92:0/1600251028 learned_addr learned my addr 172.21.15.92:0/1600251028 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f805d400640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f806814e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8066800640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f8068151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 -- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 msgr2=0x7f8068151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f8068151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 -- 172.21.15.92:0/1600251028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 msgr2=0x7f806814e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f806814e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 -- 172.21.15.92:0/1600251028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80680ca160 con 0x7f80680b34c0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.107+0000 7f8065e00640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f805800e8a0 tx=0x7f805800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.108+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058017070 con 0x7f80680b34c0 2024-03-19T02:55:08.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.108+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805800c940 con 0x7f80680b34c0 2024-03-19T02:55:08.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.108+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80680c8390 con 0x7f80680b34c0 2024-03-19T02:55:08.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.108+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805800cd30 con 0x7f80680b34c0 2024-03-19T02:55:08.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.108+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806815d350 con 0x7f80680b34c0 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.109+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806804f580 con 0x7f80680b34c0 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.110+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f805801a070 con 0x7f80680b34c0 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.110+0000 7f805ea00640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 0x7f804406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.111+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f805808a930 con 0x7f80680b34c0 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.111+0000 7f805d400640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 0x7f804406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.111+0000 7f805d400640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 0x7f804406e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8068154b10 tx=0x7f8050009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.116 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.115+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f80680ca160 con 0x7f80680b34c0 2024-03-19T02:55:08.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.344+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f806806cc00 con 0x7f80680b34c0 2024-03-19T02:55:08.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.345+0000 7f805ea00640 1 -- 172.21.15.92:0/1600251028 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f8058058d30 con 0x7f80680b34c0 2024-03-19T02:55:08.347 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:08.347 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:55:08.347 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:55:08.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 msgr2=0x7f804406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 0x7f804406e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8068154b10 tx=0x7f8050009040 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 msgr2=0x7f80680c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f805800e8a0 tx=0x7f805800ed60 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 shutdown_connections 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f804406bbd0 0x7f804406e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8068154de0 0x7f8068151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80680cd1c0 0x7f806814e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 --2- 172.21.15.92:0/1600251028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80680b34c0 0x7f80680c7c20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.348+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 >> 172.21.15.92:0/1600251028 conn(0x7f8068067130 msgr2=0x7f8068152f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:08.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.349+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 shutdown_connections 2024-03-19T02:55:08.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.349+0000 7f8067200640 1 -- 172.21.15.92:0/1600251028 wait complete. 2024-03-19T02:55:08.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:08.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.563+0000 7f0e01800640 1 Processor -- start 2024-03-19T02:55:08.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.564+0000 7f0e01800640 1 -- start start 2024-03-19T02:55:08.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0e01800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0dfc154de0 0x7f0dfc0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0e01800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0dfc0cd1c0 0x7f0dfc0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0e01800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc0b3290 con 0x7f0dfc0b34c0 2024-03-19T02:55:08.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0e01800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc0cdb30 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0e01800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc0cccc0 con 0x7f0dfc154de0 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0dfb400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f0dfc0b34c0 0x7f0dfc06bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52398/0 (socket says 172.21.15.92:52398) 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0dfb400640 1 -- 172.21.15.92:0/804486275 learned_addr learned my addr 172.21.15.92:0/804486275 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.565+0000 7f0dfbe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0dfc154de0 0x7f0dfc0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0e00e00640 1 --2- 172.21.15.92:0/804486275 >> v2:172.21.15.92:3300/0 conn(0x7f0dfc0cd1c0 0x7f0dfc0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfaa00640 1 -- 172.21.15.92:0/804486275 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428433210 0 0) 0x7f0dfc0b3290 con 0x7f0dfc0b34c0 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfaa00640 1 -- 172.21.15.92:0/804486275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ddc003520 con 0x7f0dfc0b34c0 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfbe00640 1 -- 172.21.15.92:0/804486275 >> v2:172.21.15.92:3300/0 conn(0x7f0dfc0cd1c0 msgr2=0x7f0dfc0cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfbe00640 1 --2- 172.21.15.92:0/804486275 >> v2:172.21.15.92:3300/0 conn(0x7f0dfc0cd1c0 0x7f0dfc0cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfbe00640 1 -- 172.21.15.92:0/804486275 >> v1:172.21.15.159:6789/0 conn(0x7f0dfc0b34c0 legacy=0x7f0dfc06bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.566+0000 7f0dfbe00640 1 -- 172.21.15.92:0/804486275 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc0b2cf0 con 0x7f0dfc154de0 2024-03-19T02:55:08.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.567+0000 7f0dfbe00640 1 --2- 172.21.15.92:0/804486275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc154de0 0x7f0dfc0cb170 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc067ef0 tx=0x7f0dec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb3d69298c936fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.567+0000 7f0dfaa00640 1 -- 172.21.15.92:0/804486275 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec035020 con 0x7f0dfc154de0 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0dfaa00640 1 -- 172.21.15.92:0/804486275 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dec031830 con 0x7f0dfc154de0 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0dfaa00640 1 -- 172.21.15.92:0/804486275 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec031c00 con 0x7f0dfc154de0 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 -- 172.21.15.92:0/804486275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc154de0 msgr2=0x7f0dfc0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 --2- 172.21.15.92:0/804486275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc154de0 0x7f0dfc0cb170 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc067ef0 tx=0x7f0dec02b8d0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 -- 172.21.15.92:0/804486275 shutdown_connections 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 --2- 172.21.15.92:0/804486275 >> v2:172.21.15.92:3300/0 conn(0x7f0dfc0cd1c0 0x7f0dfc0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 --2- 172.21.15.92:0/804486275 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc154de0 0x7f0dfc0cb170 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 -- 172.21.15.92:0/804486275 >> 172.21.15.92:0/804486275 conn(0x7f0dfc067130 msgr2=0x7f0dfc067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 -- 172.21.15.92:0/804486275 shutdown_connections 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.568+0000 7f0e01800640 1 -- 172.21.15.92:0/804486275 wait complete. 2024-03-19T02:55:08.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.570+0000 7f0e01800640 1 Processor -- start 2024-03-19T02:55:08.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.570+0000 7f0e01800640 1 -- start start 2024-03-19T02:55:08.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc0b34c0 0x7f0dfc0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 0x7f0dfc1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc15e1d0 con 0x7f0dfc0b34c0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0b2fb0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e01800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc15e4b0 con 0x7f0dfc154de0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0dfbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc0b34c0 0x7f0dfc0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0dfb400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0e00e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 0x7f0dfc1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0dfb400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36696/0 (socket says 172.21.15.92:36696) 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.571+0000 7f0dfb400640 1 -- 172.21.15.92:0/3722920287 learned_addr learned my addr 172.21.15.92:0/3722920287 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 -- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 msgr2=0x7f0dfc1f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 0x7f0dfc1f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 -- 172.21.15.92:0/3722920287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc0b34c0 msgr2=0x7f0dfc0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc0b34c0 0x7f0dfc0c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 -- 172.21.15.92:0/3722920287 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc0c6550 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0e00e00640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 0x7f0dfc1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0dfb400640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0de400cb40 tx=0x7f0de400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4017070 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de400aca0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4012620 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc0c59e0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.572+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc0c67f0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.573+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc04f580 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.574+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0de401a070 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.575+0000 7f0df3e00640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 0x7f0dd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.575+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0de408a7c0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.575+0000 7f0dfbe00640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 0x7f0dd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.576+0000 7f0dfbe00640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 0x7f0dd806e0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0dec036000 tx=0x7f0dec03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.579+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0de4058c50 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.767+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dfc0b3f10 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.769+0000 7f0df3e00640 1 -- 172.21.15.92:0/3722920287 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f0de405ccf0 con 0x7f0dfc0cd1c0 2024-03-19T02:55:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:08.770 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.770+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 msgr2=0x7f0dd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.770+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 0x7f0dd806e0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0dec036000 tx=0x7f0dec03c040 comp rx=0 tx=0).stop 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.770+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 msgr2=0x7f0dfc0c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.770+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0de400cb40 tx=0x7f0de400a060 comp rx=0 tx=0).stop 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 shutdown_connections 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0dd806bc00 0x7f0dd806e0c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0dfc154de0 0x7f0dfc1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0dfc0cd1c0 0x7f0dfc0c9b70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 --2- 172.21.15.92:0/3722920287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc0b34c0 0x7f0dfc0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 >> 172.21.15.92:0/3722920287 conn(0x7f0dfc067130 msgr2=0x7f0dfc0b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:08.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 shutdown_connections 2024-03-19T02:55:08.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.771+0000 7f0e01800640 1 -- 172.21.15.92:0/3722920287 wait complete. 2024-03-19T02:55:08.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:55:08.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:55:08.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.987+0000 7f606be00640 1 Processor -- start 2024-03-19T02:55:08.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.988+0000 7f606be00640 1 -- start start 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606be00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f606c0d0660 0x7f606c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 0x7f606c0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606be00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606c0c8830 con 0x7f606c0d0a40 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f606c0c79d0 con 0x7f606c0c7e60 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606be00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f606c0ccf40 con 0x7f606c0d0660 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606aa00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f606c0d0660 0x7f606c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606a000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f606c0d0a40 0x7f606c0cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52408/0 (socket says 172.21.15.92:52408) 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606a000640 1 -- 172.21.15.92:0/2707675235 learned_addr learned my addr 172.21.15.92:0/2707675235 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606b400640 1 --2- 172.21.15.92:0/2707675235 >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 0x7f606c0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606aa00640 1 -- 172.21.15.92:0/2707675235 >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 msgr2=0x7f606c0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f6069600640 1 -- 172.21.15.92:0/2707675235 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341949469 0 0) 0x7f606c0c8830 con 0x7f606c0d0a40 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606aa00640 1 --2- 172.21.15.92:0/2707675235 >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 0x7f606c0c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606aa00640 1 -- 172.21.15.92:0/2707675235 >> v1:172.21.15.159:6789/0 conn(0x7f606c0d0a40 legacy=0x7f606c0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606aa00640 1 -- 172.21.15.92:0/2707675235 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606c0cdd10 con 0x7f606c0d0660 2024-03-19T02:55:08.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.989+0000 7f606b400640 1 --2- 172.21.15.92:0/2707675235 >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 0x7f606c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.990+0000 7f606aa00640 1 --2- 172.21.15.92:0/2707675235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0660 0x7f606c0cca00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f605c0097a0 tx=0x7f605c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ae7939fc77be980 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f6069600640 1 -- 172.21.15.92:0/2707675235 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c035020 con 0x7f606c0d0660 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f6069600640 1 -- 172.21.15.92:0/2707675235 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c031830 con 0x7f606c0d0660 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f6069600640 1 -- 172.21.15.92:0/2707675235 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c031c00 con 0x7f606c0d0660 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 -- 172.21.15.92:0/2707675235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0660 msgr2=0x7f606c0cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 --2- 172.21.15.92:0/2707675235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0660 0x7f606c0cca00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f605c0097a0 tx=0x7f605c02b8d0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 -- 172.21.15.92:0/2707675235 shutdown_connections 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 --2- 172.21.15.92:0/2707675235 >> v2:172.21.15.92:3300/0 conn(0x7f606c0c7e60 0x7f606c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 --2- 172.21.15.92:0/2707675235 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0660 0x7f606c0cca00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 -- 172.21.15.92:0/2707675235 >> 172.21.15.92:0/2707675235 conn(0x7f606c072b50 msgr2=0x7f606c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 -- 172.21.15.92:0/2707675235 shutdown_connections 2024-03-19T02:55:08.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.991+0000 7f606be00640 1 -- 172.21.15.92:0/2707675235 wait complete. 2024-03-19T02:55:08.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.993+0000 7f606be00640 1 Processor -- start 2024-03-19T02:55:08.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.993+0000 7f606be00640 1 -- start start 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f606c0d0660 0x7f606c1e5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0a40 0x7f606c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f606c0cd750 con 0x7f606c0d0660 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f606c0cd470 con 0x7f606c0d0a40 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606be00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f606c0cda30 con 0x7f606c0c7e60 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606aa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606aa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49724/0 (socket says 172.21.15.92:49724) 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606aa00640 1 -- 172.21.15.92:0/3930918715 learned_addr learned my addr 172.21.15.92:0/3930918715 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.994+0000 7f606a000640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f606c0d0660 0x7f606c1e5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606b400640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0a40 0x7f606c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 -- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0a40 msgr2=0x7f606c1554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.996 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0a40 0x7f606c1554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 -- 172.21.15.92:0/3930918715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f606c0d0660 msgr2=0x7f606c1e5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:08.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f606c0d0660 0x7f606c1e5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:08.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 -- 172.21.15.92:0/3930918715 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606c1e4520 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606aa00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f605c009890 tx=0x7f605c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c041070 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c03b920 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606c1e73b0 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.995+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c035040 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.996+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f606c1e7970 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.997+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606c150100 con 0x7f606c0c7e60 2024-03-19T02:55:08.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.997+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f605c04c060 con 0x7f606c0c7e60 2024-03-19T02:55:08.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.998+0000 7f6062a00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 0x7f604806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:08.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.998+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f605c0ac410 con 0x7f606c0c7e60 2024-03-19T02:55:09.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.998+0000 7f606a000640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 0x7f604806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:08.999+0000 7f606a000640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 0x7f604806e0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f606c155040 tx=0x7f6054001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.003+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f606c150100 con 0x7f606c0c7e60 2024-03-19T02:55:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.201+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f606c04f580 con 0x7f606c0c7e60 2024-03-19T02:55:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.202+0000 7f6062a00640 1 -- 172.21.15.92:0/3930918715 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f605c07ae70 con 0x7f606c0c7e60 2024-03-19T02:55:09.203 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:55:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.204+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 msgr2=0x7f604806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.204+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 0x7f604806e0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f606c155040 tx=0x7f6054001630 comp rx=0 tx=0).stop 2024-03-19T02:55:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.204+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 msgr2=0x7f606c1e53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.204+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f605c009890 tx=0x7f605c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 shutdown_connections 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f604806bc00 0x7f604806e0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f606c0d0a40 0x7f606c1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f606c0d0660 0x7f606c1e5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 --2- 172.21.15.92:0/3930918715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f606c0c7e60 0x7f606c1e53c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 >> 172.21.15.92:0/3930918715 conn(0x7f606c072b50 msgr2=0x7f606c153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 shutdown_connections 2024-03-19T02:55:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.205+0000 7f606be00640 1 -- 172.21.15.92:0/3930918715 wait complete. 2024-03-19T02:55:09.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:55:09.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.427+0000 7f1e74000640 1 Processor -- start 2024-03-19T02:55:09.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.427+0000 7f1e74000640 1 -- start start 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e74000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 0x7f1e6c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e74000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1e6c1593d0 0x7f1e6c0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e74000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e6c0c9a80 con 0x7f1e6c0c6470 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e74000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6c0c6c90 con 0x7f1e6c1593d0 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e74000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c159910 con 0x7f1e6c152bf0 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e71600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 0x7f1e6c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e70c00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1e6c0c6470 0x7f1e6c0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52412/0 (socket says 172.21.15.92:52412) 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e70c00640 1 -- 172.21.15.92:0/649277006 learned_addr learned my addr 172.21.15.92:0/649277006 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e72000640 1 --2- 172.21.15.92:0/649277006 >> v2:172.21.15.159:3300/0 conn(0x7f1e6c1593d0 0x7f1e6c0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e71600640 1 -- 172.21.15.92:0/649277006 >> v2:172.21.15.159:3300/0 conn(0x7f1e6c1593d0 msgr2=0x7f1e6c0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.429+0000 7f1e71600640 1 --2- 172.21.15.92:0/649277006 >> v2:172.21.15.159:3300/0 conn(0x7f1e6c1593d0 0x7f1e6c0c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.430+0000 7f1e71600640 1 -- 172.21.15.92:0/649277006 >> v1:172.21.15.159:6789/0 conn(0x7f1e6c0c6470 legacy=0x7f1e6c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.430+0000 7f1e71600640 1 -- 172.21.15.92:0/649277006 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c0c8650 con 0x7f1e6c152bf0 2024-03-19T02:55:09.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.430+0000 7f1e6be00640 1 -- 172.21.15.92:0/649277006 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192373381 0 0) 0x7f1e6c0c9a80 con 0x7f1e6c0c6470 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e71600640 1 --2- 172.21.15.92:0/649277006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 0x7f1e6c06eef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e600097a0 tx=0x7f1e6002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ff3beda572159dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e6be00640 1 -- 172.21.15.92:0/649277006 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60031780 con 0x7f1e6c152bf0 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e6be00640 1 -- 172.21.15.92:0/649277006 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e60037070 con 0x7f1e6c152bf0 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e6be00640 1 -- 172.21.15.92:0/649277006 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60031b40 con 0x7f1e6c152bf0 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 -- 172.21.15.92:0/649277006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 msgr2=0x7f1e6c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 --2- 172.21.15.92:0/649277006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 0x7f1e6c06eef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e600097a0 tx=0x7f1e6002b820 comp rx=0 tx=0).stop 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 -- 172.21.15.92:0/649277006 shutdown_connections 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 --2- 172.21.15.92:0/649277006 >> v2:172.21.15.159:3300/0 conn(0x7f1e6c1593d0 0x7f1e6c0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 --2- 172.21.15.92:0/649277006 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c152bf0 0x7f1e6c06eef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 -- 172.21.15.92:0/649277006 >> 172.21.15.92:0/649277006 conn(0x7f1e6c0c8d90 msgr2=0x7f1e6c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 -- 172.21.15.92:0/649277006 shutdown_connections 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.431+0000 7f1e74000640 1 -- 172.21.15.92:0/649277006 wait complete. 2024-03-19T02:55:09.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.433+0000 7f1e74000640 1 Processor -- start 2024-03-19T02:55:09.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.433+0000 7f1e74000640 1 -- start start 2024-03-19T02:55:09.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 0x7f1e6c141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 0x7f1e6c14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 0x7f1e6c140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c06ea60 con 0x7f1e6c152bf0 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c06e4a0 con 0x7f1e6c0c6470 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e74000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c06e6f0 con 0x7f1e6c1593d0 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e71600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 0x7f1e6c141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e70c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 0x7f1e6c14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e70c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 0x7f1e6c14dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42490/0 (socket says 172.21.15.92:42490) 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e70c00640 1 -- 172.21.15.92:0/3343312500 learned_addr learned my addr 172.21.15.92:0/3343312500 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e71600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 0x7f1e6c141710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36760/0 (socket says 172.21.15.92:36760) 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.434+0000 7f1e72000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 0x7f1e6c140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 -- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 msgr2=0x7f1e6c141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.438 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 0x7f1e6c141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 -- 172.21.15.92:0/3343312500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 msgr2=0x7f1e6c14dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 0x7f1e6c14dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 -- 172.21.15.92:0/3343312500 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c0ca140 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e72000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 0x7f1e6c140b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400eba0 tx=0x7f1e6400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64018020 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6400cd20 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6c158e60 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.435+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640146c0 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.436+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6c140390 con 0x7f1e6c1593d0 2024-03-19T02:55:09.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.437+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6c0c83d0 con 0x7f1e6c1593d0 2024-03-19T02:55:09.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.437+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1e64005d10 con 0x7f1e6c1593d0 2024-03-19T02:55:09.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.438+0000 7f1e69600640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 0x7f1e4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.438+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1e6408b650 con 0x7f1e6c1593d0 2024-03-19T02:55:09.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.438+0000 7f1e71600640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 0x7f1e4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.439+0000 7f1e71600640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 0x7f1e4c06e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e60036000 tx=0x7f1e6003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.444+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1e6c0c83d0 con 0x7f1e6c1593d0 2024-03-19T02:55:09.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.638+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1e6c148c90 con 0x7f1e6c1593d0 2024-03-19T02:55:09.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.639+0000 7f1e69600640 1 -- 172.21.15.92:0/3343312500 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f1e64022030 con 0x7f1e6c1593d0 2024-03-19T02:55:09.640 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T02:55:09.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.641+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 msgr2=0x7f1e4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.641+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 0x7f1e4c06e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e60036000 tx=0x7f1e6003d040 comp rx=0 tx=0).stop 2024-03-19T02:55:09.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.641+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 msgr2=0x7f1e6c140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.641+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 0x7f1e6c140b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400eba0 tx=0x7f1e6400c490 comp rx=0 tx=0).stop 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 shutdown_connections 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1e4c06bbd0 0x7f1e4c06e090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e6c1593d0 0x7f1e6c140b50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e6c152bf0 0x7f1e6c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 --2- 172.21.15.92:0/3343312500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e6c0c6470 0x7f1e6c141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 >> 172.21.15.92:0/3343312500 conn(0x7f1e6c0c8d90 msgr2=0x7f1e6c151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:09.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 shutdown_connections 2024-03-19T02:55:09.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.642+0000 7f1e74000640 1 -- 172.21.15.92:0/3343312500 wait complete. 2024-03-19T02:55:09.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:55:09.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.871+0000 7fa144600640 1 Processor -- start 2024-03-19T02:55:09.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.871+0000 7fa144600640 1 -- start start 2024-03-19T02:55:09.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa144600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa13c1593d0 0x7fa13c0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa144600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13c0c9a80 con 0x7fa13c0c6470 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa144600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13c0c8650 con 0x7fa13c152bf0 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa144600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa13c0c6c90 con 0x7fa13c1593d0 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa141c00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa13c152bf0 0x7fa13c06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52418/0 (socket says 172.21.15.92:52418) 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa141c00640 1 -- 172.21.15.92:0/1633589110 learned_addr learned my addr 172.21.15.92:0/1633589110 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa142600640 1 --2- 172.21.15.92:0/1633589110 >> v2:172.21.15.92:3300/0 conn(0x7fa13c1593d0 0x7fa13c0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa13be00640 1 -- 172.21.15.92:0/1633589110 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395352387 0 0) 0x7fa13c0c8650 con 0x7fa13c152bf0 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa13be00640 1 -- 172.21.15.92:0/1633589110 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa124003520 con 0x7fa13c152bf0 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa142600640 1 -- 172.21.15.92:0/1633589110 >> v1:172.21.15.159:6789/0 conn(0x7fa13c152bf0 legacy=0x7fa13c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa142600640 1 -- 172.21.15.92:0/1633589110 >> v1:172.21.15.92:6789/0 conn(0x7fa13c0c6470 legacy=0x7fa13c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.873+0000 7fa142600640 1 -- 172.21.15.92:0/1633589110 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c0c8930 con 0x7fa13c1593d0 2024-03-19T02:55:09.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.874+0000 7fa142600640 1 --2- 172.21.15.92:0/1633589110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0c4690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa134009d90 tx=0x7fa13402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4feeb98b43f55d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa13be00640 1 -- 172.21.15.92:0/1633589110 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134039070 con 0x7fa13c1593d0 2024-03-19T02:55:09.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa13be00640 1 -- 172.21.15.92:0/1633589110 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa13402cc60 con 0x7fa13c1593d0 2024-03-19T02:55:09.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa13be00640 1 -- 172.21.15.92:0/1633589110 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134034660 con 0x7fa13c1593d0 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 -- 172.21.15.92:0/1633589110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 msgr2=0x7fa13c0c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 --2- 172.21.15.92:0/1633589110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0c4690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa134009d90 tx=0x7fa13402beb0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 -- 172.21.15.92:0/1633589110 shutdown_connections 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 --2- 172.21.15.92:0/1633589110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0c4690 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 -- 172.21.15.92:0/1633589110 >> 172.21.15.92:0/1633589110 conn(0x7fa13c0c8d90 msgr2=0x7fa13c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:09.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.875+0000 7fa144600640 1 -- 172.21.15.92:0/1633589110 shutdown_connections 2024-03-19T02:55:09.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.876+0000 7fa144600640 1 -- 172.21.15.92:0/1633589110 wait complete. 2024-03-19T02:55:09.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.877+0000 7fa144600640 1 Processor -- start 2024-03-19T02:55:09.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.877+0000 7fa144600640 1 -- start start 2024-03-19T02:55:09.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa13c0c6470 0x7fa13c0728c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c06e4a0 con 0x7fa13c1593d0 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c159910 con 0x7fa13c0c6470 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa144600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa13c06ea60 con 0x7fa13c152bf0 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa141200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa141c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa13c0c6470 0x7fa13c0728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa141200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49742/0 (socket says 172.21.15.92:49742) 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa141200640 1 -- 172.21.15.92:0/2710587458 learned_addr learned my addr 172.21.15.92:0/2710587458 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.878+0000 7fa142600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 -- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa13c0c6470 msgr2=0x7fa13c0728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa13c0c6470 0x7fa13c0728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 -- 172.21.15.92:0/2710587458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 msgr2=0x7fa13c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:09.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 -- 172.21.15.92:0/2710587458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c071310 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa141200640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c400 tx=0x7fa12800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1280052d0 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa128005490 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128007260 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13c0715f0 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.879+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13c0caea0 con 0x7fa13c152bf0 2024-03-19T02:55:09.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.880+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13c0ca9b0 con 0x7fa13c152bf0 2024-03-19T02:55:09.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.881+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa128007400 con 0x7fa13c152bf0 2024-03-19T02:55:09.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.882+0000 7fa139600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:09.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.882+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa128089f80 con 0x7fa13c152bf0 2024-03-19T02:55:09.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.882+0000 7fa141c00640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:09.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.883+0000 7fa141c00640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 0x7fa11c06e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa1300011e0 tx=0x7fa130007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:09.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:09.887+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa128058380 con 0x7fa13c152bf0 2024-03-19T02:55:10.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.068+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa13c0684c0 con 0x7fa13c152bf0 2024-03-19T02:55:10.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.069+0000 7fa139600640 1 -- 172.21.15.92:0/2710587458 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fa12805c420 con 0x7fa13c152bf0 2024-03-19T02:55:10.071 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T02:55:10.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 msgr2=0x7fa11c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 0x7fa11c06e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa1300011e0 tx=0x7fa130007d30 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 msgr2=0x7fa13c0cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c400 tx=0x7fa12800c8c0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 shutdown_connections 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa13c1593d0 0x7fa13c0cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa13c152bf0 0x7fa13c0cb6a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 --2- 172.21.15.92:0/2710587458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa13c0c6470 0x7fa13c0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 >> 172.21.15.92:0/2710587458 conn(0x7fa13c0c8d90 msgr2=0x7fa13c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 shutdown_connections 2024-03-19T02:55:10.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.072+0000 7fa144600640 1 -- 172.21.15.92:0/2710587458 wait complete. 2024-03-19T02:55:10.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:10.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.296+0000 7ff467600640 1 Processor -- start 2024-03-19T02:55:10.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.296+0000 7ff467600640 1 -- start start 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.297+0000 7ff467600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff4600c60d0 0x7ff460071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.297+0000 7ff467600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff46015b740 con 0x7ff460072380 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.297+0000 7ff467600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff46015ba20 con 0x7ff460154dc0 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.297+0000 7ff467600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff4600c8400 con 0x7ff4600c60d0 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff457e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff460072380 0x7ff4600c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46368/0 (socket says 172.21.15.92:46368) 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff457e00640 1 -- 172.21.15.92:0/3525908260 learned_addr learned my addr 172.21.15.92:0/3525908260 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff464c00640 1 --1- 172.21.15.92:0/3525908260 >> v1:172.21.15.159:6789/0 conn(0x7ff460154dc0 0x7ff460071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52420/0 (socket says 172.21.15.92:52420) 2024-03-19T02:55:10.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff465600640 1 --2- 172.21.15.92:0/3525908260 >> v2:172.21.15.159:3300/0 conn(0x7ff4600c60d0 0x7ff460071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950029743 0 0) 0x7ff46015ba20 con 0x7ff460154dc0 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff444003520 con 0x7ff460154dc0 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901666911 0 0) 0x7ff46015b740 con 0x7ff460072380 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff465600640 1 -- 172.21.15.92:0/3525908260 >> v1:172.21.15.159:6789/0 conn(0x7ff460154dc0 legacy=0x7ff460071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff465600640 1 -- 172.21.15.92:0/3525908260 >> v1:172.21.15.92:6789/0 conn(0x7ff460072380 legacy=0x7ff4600c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.298+0000 7ff465600640 1 -- 172.21.15.92:0/3525908260 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4600c86e0 con 0x7ff4600c60d0 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.299+0000 7ff465600640 1 --2- 172.21.15.92:0/3525908260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4600c60d0 0x7ff460071190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff458009d90 tx=0x7ff45802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b252a708d760a7b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.299+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458039070 con 0x7ff4600c60d0 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.299+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff45802cc60 con 0x7ff4600c60d0 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.299+0000 7ff45fe00640 1 -- 172.21.15.92:0/3525908260 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458034640 con 0x7ff4600c60d0 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 -- 172.21.15.92:0/3525908260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4600c60d0 msgr2=0x7ff460071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 --2- 172.21.15.92:0/3525908260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4600c60d0 0x7ff460071190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff458009d90 tx=0x7ff45802beb0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 -- 172.21.15.92:0/3525908260 shutdown_connections 2024-03-19T02:55:10.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 --2- 172.21.15.92:0/3525908260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4600c60d0 0x7ff460071190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 -- 172.21.15.92:0/3525908260 >> 172.21.15.92:0/3525908260 conn(0x7ff4600c8d90 msgr2=0x7ff4600c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:10.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 -- 172.21.15.92:0/3525908260 shutdown_connections 2024-03-19T02:55:10.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.300+0000 7ff467600640 1 -- 172.21.15.92:0/3525908260 wait complete. 2024-03-19T02:55:10.303 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.302+0000 7ff467600640 1 Processor -- start 2024-03-19T02:55:10.303 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.302+0000 7ff467600640 1 -- start start 2024-03-19T02:55:10.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff460072380 0x7ff46014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4600c60d0 0x7ff46014db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600c7520 con 0x7ff4600c60d0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600c73a0 con 0x7ff460072380 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff467600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff4600c76a0 con 0x7ff460154dc0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff457e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4600c60d0 0x7ff46014db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49756/0 (socket says 172.21.15.92:49756) 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 -- 172.21.15.92:0/1728740874 learned_addr learned my addr 172.21.15.92:0/1728740874 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff464c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff460072380 0x7ff46014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 -- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff460072380 msgr2=0x7ff46014f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff460072380 0x7ff46014f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 -- 172.21.15.92:0/1728740874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4600c60d0 msgr2=0x7ff46014db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4600c60d0 0x7ff46014db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 -- 172.21.15.92:0/1728740874 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff460148980 con 0x7ff460154dc0 2024-03-19T02:55:10.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.303+0000 7ff465600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff458009eb0 tx=0x7ff458004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.304+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458039050 con 0x7ff460154dc0 2024-03-19T02:55:10.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.304+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff458045070 con 0x7ff460154dc0 2024-03-19T02:55:10.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.304+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45803e830 con 0x7ff460154dc0 2024-03-19T02:55:10.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.304+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff460148c60 con 0x7ff460154dc0 2024-03-19T02:55:10.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.304+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4600cb270 con 0x7ff460154dc0 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.305+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4600c6980 con 0x7ff460154dc0 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.306+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff45803e9d0 con 0x7ff460154dc0 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.306+0000 7ff45d600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 0x7ff43c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.307+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff4580ad710 con 0x7ff460154dc0 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.307+0000 7ff464c00640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 0x7ff43c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.307+0000 7ff464c00640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 0x7ff43c06e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4500011e0 tx=0x7ff450007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.311+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4600c6980 con 0x7ff460154dc0 2024-03-19T02:55:10.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.545+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff460068de0 con 0x7ff460154dc0 2024-03-19T02:55:10.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.546+0000 7ff45d600640 1 -- 172.21.15.92:0/1728740874 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7ff460068de0 con 0x7ff460154dc0 2024-03-19T02:55:10.548 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:10.548 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:55:10.548 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:55:10.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.548+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 msgr2=0x7ff43c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.548+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 0x7ff43c06e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4500011e0 tx=0x7ff450007d30 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 msgr2=0x7ff460158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff458009eb0 tx=0x7ff458004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 shutdown_connections 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff43c06bbd0 0x7ff43c06e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff460154dc0 0x7ff460158c70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4600c60d0 0x7ff46014db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 --2- 172.21.15.92:0/1728740874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff460072380 0x7ff46014f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 >> 172.21.15.92:0/1728740874 conn(0x7ff4600c8d90 msgr2=0x7ff46006e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 shutdown_connections 2024-03-19T02:55:10.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.549+0000 7ff467600640 1 -- 172.21.15.92:0/1728740874 wait complete. 2024-03-19T02:55:10.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:10.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.763+0000 7f1c3d600640 1 Processor -- start 2024-03-19T02:55:10.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.763+0000 7f1c3d600640 1 -- start start 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1c38154dc0 0x7f1c38071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c380721b0 0x7f1c380c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1c38072660 con 0x7f1c38154dc0 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3815b740 con 0x7f1c380721b0 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c3d600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3815ba20 con 0x7f1c380c60d0 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c2e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c2e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49762/0 (socket says 172.21.15.92:49762) 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c2e000640 1 -- 172.21.15.92:0/3396340124 learned_addr learned my addr 172.21.15.92:0/3396340124 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c37400640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.159:3300/0 conn(0x7f1c380721b0 0x7f1c380c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.765+0000 7f1c36a00640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.92:3300/0 conn(0x7f1c38154dc0 0x7f1c38071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.766+0000 7f1c2e000640 1 -- 172.21.15.92:0/3396340124 >> v2:172.21.15.159:3300/0 conn(0x7f1c380721b0 msgr2=0x7f1c380c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.766+0000 7f1c2e000640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.159:3300/0 conn(0x7f1c380721b0 0x7f1c380c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.766+0000 7f1c2e000640 1 -- 172.21.15.92:0/3396340124 >> v2:172.21.15.92:3300/0 conn(0x7f1c38154dc0 msgr2=0x7f1c38071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.766+0000 7f1c2e000640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.92:3300/0 conn(0x7f1c38154dc0 0x7f1c38071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.766+0000 7f1c2e000640 1 -- 172.21.15.92:0/3396340124 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c380c8400 con 0x7f1c380c60d0 2024-03-19T02:55:10.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c2e000640 1 --2- 172.21.15.92:0/3396340124 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c3806e240 tx=0x7f1c2002b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a5facbcb50e1958 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c36000640 1 -- 172.21.15.92:0/3396340124 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200317f0 con 0x7f1c380c60d0 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c36000640 1 -- 172.21.15.92:0/3396340124 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c20037070 con 0x7f1c380c60d0 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c36000640 1 -- 172.21.15.92:0/3396340124 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20031bb0 con 0x7f1c380c60d0 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c3d600640 1 -- 172.21.15.92:0/3396340124 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 msgr2=0x7f1c38071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.767+0000 7f1c3d600640 1 --2- 172.21.15.92:0/3396340124 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c3806e240 tx=0x7f1c2002b890 comp rx=0 tx=0).stop 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 -- 172.21.15.92:0/3396340124 shutdown_connections 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.159:3300/0 conn(0x7f1c380721b0 0x7f1c380c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 --2- 172.21.15.92:0/3396340124 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c380c60d0 0x7f1c38071c70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 --2- 172.21.15.92:0/3396340124 >> v2:172.21.15.92:3300/0 conn(0x7f1c38154dc0 0x7f1c38071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 -- 172.21.15.92:0/3396340124 >> 172.21.15.92:0/3396340124 conn(0x7f1c380c8d90 msgr2=0x7f1c380c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 -- 172.21.15.92:0/3396340124 shutdown_connections 2024-03-19T02:55:10.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.768+0000 7f1c3d600640 1 -- 172.21.15.92:0/3396340124 wait complete. 2024-03-19T02:55:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.769+0000 7f1c3d600640 1 Processor -- start 2024-03-19T02:55:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.769+0000 7f1c3d600640 1 -- start start 2024-03-19T02:55:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 0x7f1c3815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 0x7f1c38155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1c380c71d0 con 0x7f1c380721b0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381551a0 con 0x7f1c380c60d0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c3d600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c380c7350 con 0x7f1c38154dc0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.770+0000 7f1c37400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c2e000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 0x7f1c38155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 0x7f1c3815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49772/0 (socket says 172.21.15.92:49772) 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 -- 172.21.15.92:0/360194135 learned_addr learned my addr 172.21.15.92:0/360194135 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 -- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 msgr2=0x7f1c38155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 0x7f1c38155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 -- 172.21.15.92:0/360194135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 msgr2=0x7f1c3815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 0x7f1c3815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 -- 172.21.15.92:0/360194135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c381f0760 con 0x7f1c38154dc0 2024-03-19T02:55:10.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c2e000640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 0x7f1c38155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c36a00640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 0x7f1c3815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c37400640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000ef00 tx=0x7f1c3000b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.771+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c30018020 con 0x7f1c38154dc0 2024-03-19T02:55:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.772+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c30014400 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.772+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300147f0 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.772+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c381f0a40 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.772+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c381f1000 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.773+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c380684c0 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.774+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1c30005d10 con 0x7f1c38154dc0 2024-03-19T02:55:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.774+0000 7f1c2f400640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 0x7f1c1406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.774+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1c3008b130 con 0x7f1c38154dc0 2024-03-19T02:55:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.775+0000 7f1c36a00640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 0x7f1c1406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.775+0000 7f1c36a00640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 0x7f1c1406e0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c28007d90 tx=0x7f1c28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:10.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.779+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1c30022070 con 0x7f1c38154dc0 2024-03-19T02:55:10.899 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-03-19T02:55:10.898+0000 7fcb875e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T02:55:10.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.962+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c380c6e70 con 0x7f1c38154dc0 2024-03-19T02:55:10.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.964+0000 7f1c2f400640 1 -- 172.21.15.92:0/360194135 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f1c300594a0 con 0x7f1c38154dc0 2024-03-19T02:55:10.965 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:10.965 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:10.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 msgr2=0x7f1c1406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 0x7f1c1406e0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c28007d90 tx=0x7f1c28008040 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 msgr2=0x7f1c38155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000ef00 tx=0x7f1c3000b480 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 shutdown_connections 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1c1406bc00 0x7f1c1406e0c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38154dc0 0x7f1c38155f70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380c60d0 0x7f1c38155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 --2- 172.21.15.92:0/360194135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c380721b0 0x7f1c3815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 >> 172.21.15.92:0/360194135 conn(0x7f1c380c8d90 msgr2=0x7f1c380c50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 shutdown_connections 2024-03-19T02:55:10.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:10.966+0000 7f1c3d600640 1 -- 172.21.15.92:0/360194135 wait complete. 2024-03-19T02:55:10.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:11.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.186+0000 7efcf3200640 1 Processor -- start 2024-03-19T02:55:11.187 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.186+0000 7efcf3200640 1 -- start start 2024-03-19T02:55:11.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.187+0000 7efcf3200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efcec154dc0 0x7efcec071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:11.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcf3200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcec15b740 con 0x7efcec0c60d0 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcf3200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcec15ba20 con 0x7efcec072380 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcf3200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7efcec0c8400 con 0x7efcec154dc0 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcf1200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efcec072380 0x7efcec0c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52430/0 (socket says 172.21.15.92:52430) 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efce3400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efcec0c60d0 0x7efcec071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46370/0 (socket says 172.21.15.92:46370) 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcf1200640 1 -- 172.21.15.92:0/2049609577 learned_addr learned my addr 172.21.15.92:0/2049609577 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcebe00640 1 --2- 172.21.15.92:0/2049609577 >> v2:172.21.15.159:3300/0 conn(0x7efcec154dc0 0x7efcec071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194470427 0 0) 0x7efcec15b740 con 0x7efcec0c60d0 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcd0003520 con 0x7efcec0c60d0 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388660642 0 0) 0x7efcec15ba20 con 0x7efcec072380 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcec15b740 con 0x7efcec072380 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcebe00640 1 -- 172.21.15.92:0/2049609577 >> v1:172.21.15.159:6789/0 conn(0x7efcec072380 legacy=0x7efcec0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcebe00640 1 -- 172.21.15.92:0/2049609577 >> v1:172.21.15.92:6789/0 conn(0x7efcec0c60d0 legacy=0x7efcec071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.188+0000 7efcebe00640 1 -- 172.21.15.92:0/2049609577 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec0c86e0 con 0x7efcec154dc0 2024-03-19T02:55:11.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.189+0000 7efcebe00640 1 --2- 172.21.15.92:0/2049609577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec154dc0 0x7efcec071190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009d90 tx=0x7efcdc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ca5824559839467 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:11.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc039070 con 0x7efcec154dc0 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcdc02cc60 con 0x7efcec154dc0 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efceb400640 1 -- 172.21.15.92:0/2049609577 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc034640 con 0x7efcec154dc0 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 -- 172.21.15.92:0/2049609577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec154dc0 msgr2=0x7efcec071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 --2- 172.21.15.92:0/2049609577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec154dc0 0x7efcec071190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009d90 tx=0x7efcdc02beb0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 -- 172.21.15.92:0/2049609577 shutdown_connections 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 --2- 172.21.15.92:0/2049609577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec154dc0 0x7efcec071190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 -- 172.21.15.92:0/2049609577 >> 172.21.15.92:0/2049609577 conn(0x7efcec0c8d90 msgr2=0x7efcec0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 -- 172.21.15.92:0/2049609577 shutdown_connections 2024-03-19T02:55:11.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.190+0000 7efcf3200640 1 -- 172.21.15.92:0/2049609577 wait complete. 2024-03-19T02:55:11.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.192+0000 7efcf3200640 1 Processor -- start 2024-03-19T02:55:11.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.192+0000 7efcf3200640 1 -- start start 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec0c60d0 0x7efcec1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 0x7efcec14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efcec0c7520 con 0x7efcec154dc0 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efcec0c73a0 con 0x7efcec0c60d0 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf3200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efcec0c76a0 con 0x7efcec072380 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcf1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 0x7efcec14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcebe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcebe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49778/0 (socket says 172.21.15.92:49778) 2024-03-19T02:55:11.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efcebe00640 1 -- 172.21.15.92:0/3154719913 learned_addr learned my addr 172.21.15.92:0/3154719913 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.193+0000 7efce3400640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec0c60d0 0x7efcec1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 -- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 msgr2=0x7efcec1f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 -- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec0c60d0 msgr2=0x7efcec1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec0c60d0 0x7efcec1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 -- 172.21.15.92:0/3154719913 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec1f0660 con 0x7efcec154dc0 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcebe00640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:11.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf1200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 0x7efcec14f800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efce400c3f0 tx=0x7efce400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4017070 con 0x7efcec154dc0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce4005ce0 con 0x7efcec154dc0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcec156220 con 0x7efcec154dc0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce40072d0 con 0x7efcec154dc0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.194+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcec1f0be0 con 0x7efcec154dc0 2024-03-19T02:55:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.196+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcec0714a0 con 0x7efcec154dc0 2024-03-19T02:55:11.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.197+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efce401a070 con 0x7efcec154dc0 2024-03-19T02:55:11.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.198+0000 7efce8c00640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 0x7efcc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:11.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.198+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efce4089d70 con 0x7efcec154dc0 2024-03-19T02:55:11.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.198+0000 7efcebe00640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 0x7efcc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:11.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.198+0000 7efcebe00640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 0x7efcc806e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efcdc02bef0 tx=0x7efcdc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:11.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.202+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efcec0714a0 con 0x7efcec154dc0 2024-03-19T02:55:11.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.438+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcec0684c0 con 0x7efcec154dc0 2024-03-19T02:55:11.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.439+0000 7efce8c00640 1 -- 172.21.15.92:0/3154719913 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7efce4058170 con 0x7efcec154dc0 2024-03-19T02:55:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:11.441 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:55:11.441 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:55:11.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 msgr2=0x7efcc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 0x7efcc806e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efcdc02bef0 tx=0x7efcdc030000 comp rx=0 tx=0).stop 2024-03-19T02:55:11.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 msgr2=0x7efcec14f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 0x7efcec14f800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efce400c3f0 tx=0x7efce400c8b0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 shutdown_connections 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efcc806bbd0 0x7efcc806e090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcec154dc0 0x7efcec14f800 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efcec0c60d0 0x7efcec1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 --2- 172.21.15.92:0/3154719913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efcec072380 0x7efcec1f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 >> 172.21.15.92:0/3154719913 conn(0x7efcec0c8d90 msgr2=0x7efcec071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 shutdown_connections 2024-03-19T02:55:11.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:11.442+0000 7efcf3200640 1 -- 172.21.15.92:0/3154719913 wait complete. 2024-03-19T02:55:12.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T02:54:55.655938+0000', 'modified': '2024-03-19T02:55:02.221279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.159:6839/1521694689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6838', 'nonce': 1521694689}, {'type': 'v1', 'addr': '172.21.15.159:6839', 'nonce': 1521694689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4264, 'qdb_cluster': [4264]} 2024-03-19T02:55:12.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T02:55:12.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-19T02:55:12.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.287+0000 7f5636400640 1 Processor -- start 2024-03-19T02:55:12.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.287+0000 7f5636400640 1 -- start start 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5636400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 0x7f563015bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5636400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56300cd880 0x7f563015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5636400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56301552e0 con 0x7f56300ca2e0 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5636400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f563015b7f0 con 0x7f56300cd880 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5636400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56300ccac0 con 0x7f56300cb6b0 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5635000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f56300ca2e0 0x7f56300cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46372/0 (socket says 172.21.15.92:46372) 2024-03-19T02:55:12.290 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f562fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 0x7f563015bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.288+0000 7f5635000640 1 -- 172.21.15.92:0/1675739373 learned_addr learned my addr 172.21.15.92:0/1675739373 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.289+0000 7f5635a00640 1 --2- 172.21.15.92:0/1675739373 >> v2:172.21.15.92:3300/0 conn(0x7f56300cd880 0x7f563015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.289+0000 7f562fe00640 1 -- 172.21.15.92:0/1675739373 >> v2:172.21.15.92:3300/0 conn(0x7f56300cd880 msgr2=0x7f563015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.289+0000 7f562fe00640 1 --2- 172.21.15.92:0/1675739373 >> v2:172.21.15.92:3300/0 conn(0x7f56300cd880 0x7f563015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.289+0000 7f562fe00640 1 -- 172.21.15.92:0/1675739373 >> v1:172.21.15.92:6789/0 conn(0x7f56300ca2e0 legacy=0x7f56300cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.291 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.289+0000 7f562fe00640 1 -- 172.21.15.92:0/1675739373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563015bad0 con 0x7f56300cb6b0 2024-03-19T02:55:12.292 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.290+0000 7f562fe00640 1 --2- 172.21.15.92:0/1675739373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 0x7f563015bfb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f561c0097a0 tx=0x7f561c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb284c4518b11e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.292 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.290+0000 7f562f400640 1 -- 172.21.15.92:0/1675739373 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c035020 con 0x7f56300cb6b0 2024-03-19T02:55:12.292 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.290+0000 7f562f400640 1 -- 172.21.15.92:0/1675739373 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561c031830 con 0x7f56300cb6b0 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.290+0000 7f562f400640 1 -- 172.21.15.92:0/1675739373 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c031c00 con 0x7f56300cb6b0 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 -- 172.21.15.92:0/1675739373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 msgr2=0x7f563015bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 --2- 172.21.15.92:0/1675739373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 0x7f563015bfb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f561c0097a0 tx=0x7f561c02b8d0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 -- 172.21.15.92:0/1675739373 shutdown_connections 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 --2- 172.21.15.92:0/1675739373 >> v2:172.21.15.92:3300/0 conn(0x7f56300cd880 0x7f563015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 --2- 172.21.15.92:0/1675739373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300cb6b0 0x7f563015bfb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 -- 172.21.15.92:0/1675739373 >> 172.21.15.92:0/1675739373 conn(0x7f5630072970 msgr2=0x7f5630072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 -- 172.21.15.92:0/1675739373 shutdown_connections 2024-03-19T02:55:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.291+0000 7f5636400640 1 -- 172.21.15.92:0/1675739373 wait complete. 2024-03-19T02:55:12.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.293+0000 7f5636400640 1 Processor -- start 2024-03-19T02:55:12.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.293+0000 7f5636400640 1 -- start start 2024-03-19T02:55:12.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56300cb6b0 0x7f56301578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56300cd880 0x7f5630154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f56300cd360 con 0x7f56300cb6b0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f56300cd080 con 0x7f56300cd880 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5636400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56300cc580 con 0x7f56300ca2e0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f562fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56300cb6b0 0x7f56301578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49792/0 (socket says 172.21.15.92:49792) 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 -- 172.21.15.92:0/3401081523 learned_addr learned my addr 172.21.15.92:0/3401081523 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635a00640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56300cd880 0x7f5630154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 -- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56300cd880 msgr2=0x7f5630154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56300cd880 0x7f5630154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 -- 172.21.15.92:0/3401081523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56300cb6b0 msgr2=0x7f56301578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56300cb6b0 0x7f56301578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.294+0000 7f5635000640 1 -- 172.21.15.92:0/3401081523 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56301582f0 con 0x7f56300ca2e0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f5635000640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f562400c400 tx=0x7f562400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.297 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f562cc00640 1 -- 172.21.15.92:0/3401081523 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624017070 con 0x7f56300ca2e0 2024-03-19T02:55:12.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f562cc00640 1 -- 172.21.15.92:0/3401081523 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5624005ce0 con 0x7f56300ca2e0 2024-03-19T02:55:12.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f562cc00640 1 -- 172.21.15.92:0/3401081523 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56240072f0 con 0x7f56300ca2e0 2024-03-19T02:55:12.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56301f3e00 con 0x7f56300ca2e0 2024-03-19T02:55:12.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.295+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301f43a0 con 0x7f56300ca2e0 2024-03-19T02:55:12.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.297+0000 7f562cc00640 1 -- 172.21.15.92:0/3401081523 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f562401a070 con 0x7f56300ca2e0 2024-03-19T02:55:12.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.298+0000 7f562cc00640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 0x7f561006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.298+0000 7f562cc00640 1 -- 172.21.15.92:0/3401081523 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5624089e60 con 0x7f56300ca2e0 2024-03-19T02:55:12.300 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.299+0000 7f562fe00640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 0x7f561006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.299+0000 7f562fe00640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 0x7f561006e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f561c036000 tx=0x7f561c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.299+0000 7f563a7a4740 1 Processor -- start 2024-03-19T02:55:12.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.299+0000 7f563a7a4740 1 -- start start 2024-03-19T02:55:12.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc396450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 0x55e2bc3d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc2f9d00 con 0x55e2bc3dfef0 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc2a2b00 con 0x55e2bc396070 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55e2bc2f5da0 con 0x55e2bc394ad0 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f562fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc396450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f5635a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 0x55e2bc3d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f562fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42592/0 (socket says 172.21.15.92:42592) 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc396450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36842/0 (socket says 172.21.15.92:36842) 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.300+0000 7f562fe00640 1 -- 172.21.15.92:0/3109760540 learned_addr learned my addr 172.21.15.92:0/3109760540 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 -- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 msgr2=0x55e2bc396450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc396450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 -- 172.21.15.92:0/3109760540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 msgr2=0x55e2bc396990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 -- 172.21.15.92:0/3109760540 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2bc00e2d0 con 0x55e2bc394ad0 2024-03-19T02:55:12.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f562fe00640 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:12.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5635a00640 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 0x55e2bc3d58d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5628011360 tx=0x7f5628009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b232af28724b1f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5621600640 1 -- 172.21.15.92:0/3109760540 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628003b90 con 0x55e2bc394ad0 2024-03-19T02:55:12.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5621600640 1 -- 172.21.15.92:0/3109760540 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628003d30 con 0x55e2bc394ad0 2024-03-19T02:55:12.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.301+0000 7f5621600640 1 -- 172.21.15.92:0/3109760540 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628004240 con 0x55e2bc394ad0 2024-03-19T02:55:12.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 -- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 msgr2=0x55e2bc3d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 0x55e2bc3d58d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5628011360 tx=0x7f5628009750 comp rx=0 tx=0).stop 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 -- 172.21.15.92:0/3109760540 shutdown_connections 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc394ad0 0x55e2bc3d58d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc396990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3109760540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc396450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 -- 172.21.15.92:0/3109760540 >> 172.21.15.92:0/3109760540 conn(0x55e2bc3927e0 msgr2=0x55e2bc395120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 -- 172.21.15.92:0/3109760540 shutdown_connections 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.302+0000 7f563a7a4740 1 -- 172.21.15.92:0/3109760540 wait complete. 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.303+0000 7f563a7a4740 1 -- start start 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.303+0000 7f563a7a4740 1 Processor -- start 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3dfef0 0x55e2bc3bd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 0x55e2bc3db940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc3bb5d0 con 0x55e2bc396070 2024-03-19T02:55:12.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc2b5ed0 con 0x55e2bc3db4c0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55e2bc393990 con 0x55e2bc3dfef0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f563a7a4740 10 client.0 _ll_register_callbacks cb 0x55e2bc2b3c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f5635000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f562fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3dfef0 0x55e2bc3bd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f5635a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 0x55e2bc3db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f5635000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42598/0 (socket says 172.21.15.92:42598) 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.304+0000 7f5635000640 1 -- 172.21.15.92:0/3144865196 learned_addr learned my addr 172.21.15.92:0/3144865196 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 -- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3dfef0 msgr2=0x55e2bc3bd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3dfef0 0x55e2bc3bd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 -- 172.21.15.92:0/3144865196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 msgr2=0x55e2bc3bbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 -- 172.21.15.92:0/3144865196 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2bc0d86b0 con 0x55e2bc3db4c0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635000640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5635a00640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 0x55e2bc3db940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5628011330 tx=0x7f5628004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0d46104eb0caf15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5616000640 10 client.0 ms_handle_connect on v2:172.21.15.159:3300/0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562802d060 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5628003d30 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.305+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mon.1 v2:172.21.15.159:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5628015970 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f5628033050 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 10 client.4633 fetch_fsmap learned FSMap version 8 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e2bc3bcf50 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mon.1 v2:172.21.15.159:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628003d30 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mon.1 v2:172.21.15.159:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f5628036020 con 0x55e2bc3db4c0 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 10 client.4633 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 10 client.4633 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 20 client.4633 populate_metadata read hostname 'smithi092' 2024-03-19T02:55:12.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc3ba1a0 0x55e2bc3f6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 4 client.4633 mds_command: new command op to 4264 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 --> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2bc307240 con 0x55e2bc3ba1a0 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.306+0000 7f5635000640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc3ba1a0 0x55e2bc3f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.307+0000 7f5635000640 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc3ba1a0 0x55e2bc3f6400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a71010418310c95 server_cookie=fb64c02eb97b6e97 in_seq=0 out_seq=0 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.307+0000 7f5616000640 10 client.4633 ms_handle_connect on v2:172.21.15.159:6838/1521694689 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f5616000640 1 -- 172.21.15.92:0/3144865196 <== mds.0 v2:172.21.15.159:6838/1521694689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2bc307240 con 0x55e2bc3ba1a0 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f5616000640 10 client.4633 handle_command_reply: tid=0 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 client.4633 shutdown 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 msgr2=0x55e2bc3db940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 0x55e2bc3db940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5628011330 tx=0x7f5628004ab0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 shutdown_connections 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc3ba1a0 0x55e2bc3f6400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3db4c0 0x55e2bc3db940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3dfef0 0x55e2bc3bd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.317 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 --2- 172.21.15.92:0/3144865196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc3bbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f5616000640 0 client.4633 ms_handle_reset on v2:172.21.15.159:6838/1521694689 2024-03-19T02:55:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 >> 172.21.15.92:0/3144865196 conn(0x55e2bc394ad0 msgr2=0x55e2bc396450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 shutdown_connections 2024-03-19T02:55:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 1 -- 172.21.15.92:0/3144865196 wait complete. 2024-03-19T02:55:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.308+0000 7f563a7a4740 20 client.4633 trim_cache size 0 max 16384 2024-03-19T02:55:12.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.328+0000 7f563a7a4740 1 Processor -- start 2024-03-19T02:55:12.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.328+0000 7f563a7a4740 1 -- start start 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.328+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.328+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.328+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 0x55e2bc394fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f563a7a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5630151790 con 0x55e2bc396070 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc3d36e0 con 0x55e2bc3dfef0 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55e2bc3dbfc0 con 0x55e2bc3db4c0 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f5635000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f5635a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 0x55e2bc394fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:49822/0 (socket says 172.21.15.92:49822) 2024-03-19T02:55:12.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f5635000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42612/0 (socket says 172.21.15.92:42612) 2024-03-19T02:55:12.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 -- 172.21.15.92:0/225597056 learned_addr learned my addr 172.21.15.92:0/225597056 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 -- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 msgr2=0x55e2bc394fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 0x55e2bc394fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 -- 172.21.15.92:0/225597056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 msgr2=0x55e2bc38f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f562fe00640 1 -- 172.21.15.92:0/225597056 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2bc392e60 con 0x55e2bc3db4c0 2024-03-19T02:55:12.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f5635a00640 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 0x55e2bc394fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.329+0000 7f5635000640 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f562fe00640 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f561c031d90 tx=0x7f561c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99602c5c0c824f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f5620c00640 1 -- 172.21.15.92:0/225597056 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c035040 con 0x55e2bc3db4c0 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f5620c00640 1 -- 172.21.15.92:0/225597056 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561c0479a0 con 0x55e2bc3db4c0 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f5620c00640 1 -- 172.21.15.92:0/225597056 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c047d50 con 0x55e2bc3db4c0 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 -- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 msgr2=0x55e2bc38fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f561c031d90 tx=0x7f561c031000 comp rx=0 tx=0).stop 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 -- 172.21.15.92:0/225597056 shutdown_connections 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc3dfef0 0x55e2bc394fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc3db4c0 0x55e2bc38fa40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 --2- 172.21.15.92:0/225597056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc396070 0x55e2bc38f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 -- 172.21.15.92:0/225597056 >> 172.21.15.92:0/225597056 conn(0x55e2bc394ad0 msgr2=0x55e2bc38b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 -- 172.21.15.92:0/225597056 shutdown_connections 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.330+0000 7f563a7a4740 1 -- 172.21.15.92:0/225597056 wait complete. 2024-03-19T02:55:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.331+0000 7f563a7a4740 1 -- start start 2024-03-19T02:55:12.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.331+0000 7f563a7a4740 1 Processor -- start 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc393cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc453c90 0x55e2bc454110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc3eaa50 con 0x55e2bc3dfef0 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55e2bc3ba5c0 con 0x55e2bc396070 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55e2bc3eb0e0 con 0x55e2bc453c90 2024-03-19T02:55:12.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f563a7a4740 10 client.0 _ll_register_callbacks cb 0x7f561c009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f5635000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36862/0 (socket says 172.21.15.92:36862) 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f5635000640 1 -- 172.21.15.92:0/2925434871 learned_addr learned my addr 172.21.15.92:0/2925434871 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f562fe00640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc393cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.332+0000 7f5635a00640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc453c90 0x55e2bc454110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 -- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc453c90 msgr2=0x55e2bc454110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc453c90 0x55e2bc454110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 -- 172.21.15.92:0/2925434871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 msgr2=0x55e2bc393cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc393cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 -- 172.21.15.92:0/2925434871 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2bc3d08f0 con 0x55e2bc396070 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5635000640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5624088290 tx=0x7f5624000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4995848f480ed069 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5616000640 10 client.0 ms_handle_connect on v2:172.21.15.159:3300/0 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624007490 con 0x55e2bc396070 2024-03-19T02:55:12.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mon.1 v2:172.21.15.159:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5624007630 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f561c0479a0 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f562407c050 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f563a7a4740 10 client.4639 fetch_fsmap learned FSMap version 8 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.333+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e2bc3fb290 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mon.1 v2:172.21.15.159:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561c0479a0 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mon.1 v2:172.21.15.159:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f562407f020 con 0x55e2bc396070 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 10 client.4639 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 10 client.4639 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 20 client.4639 populate_metadata read hostname 'smithi092' 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc464870 0x55e2bc466c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 4 client.4639 mds_command: new command op to 4264 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 --> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] -- command(tid 0: {"prefix": "status"}) -- 0x55e2bc301590 con 0x55e2bc464870 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.334+0000 7f562fe00640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc464870 0x55e2bc466c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f562fe00640 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc464870 0x55e2bc466c60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56be3df7ecb53143 server_cookie=5d06002af5d37372 in_seq=0 out_seq=0 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f5616000640 10 client.4639 ms_handle_connect on v2:172.21.15.159:6838/1521694689 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f5616000640 1 -- 172.21.15.92:0/2925434871 <== mds.0 v2:172.21.15.159:6838/1521694689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55e2bc301590 con 0x55e2bc464870 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f5616000640 10 client.4639 handle_command_reply: tid=0 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 client.4639 shutdown 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 msgr2=0x55e2bc395860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5624088290 tx=0x7f5624000ed0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 shutdown_connections 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:6838/1521694689,v1:172.21.15.159:6839/1521694689] conn(0x55e2bc464870 0x55e2bc466c60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55e2bc453c90 0x55e2bc454110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e2bc3dfef0 0x55e2bc393cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f5616000640 0 client.4639 ms_handle_reset on v2:172.21.15.159:6838/1521694689 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 --2- 172.21.15.92:0/2925434871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55e2bc396070 0x55e2bc395860 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 >> 172.21.15.92:0/2925434871 conn(0x55e2bc394ad0 msgr2=0x55e2bc3bd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 shutdown_connections 2024-03-19T02:55:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 1 -- 172.21.15.92:0/2925434871 wait complete. 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.335+0000 7f563a7a4740 20 client.4639 trim_cache size 0 max 16384 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_fsid": "91932042-1f9e-4210-9e82-70c11e6277fd", 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "whoami": 0, 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "id": 4264, 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "want_state": "up:active", 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "state": "up:active", 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "fs_name": "cephfs", 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rank_uptime": 16.166767590999999, 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "mdsmap_epoch": 8, 2024-03-19T02:55:12.341 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_epoch": 20, 2024-03-19T02:55:12.342 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_epoch_barrier": 20, 2024-03-19T02:55:12.342 INFO:teuthology.orchestra.run.smithi092.stdout: "uptime": 21.450372680000001 2024-03-19T02:55:12.342 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T02:55:12.342 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 msgr2=0x7f561006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 0x7f561006e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f561c036000 tx=0x7f561c008030 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 msgr2=0x7f56301568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f562400c400 tx=0x7f562400c8c0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 shutdown_connections 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f561006bbd0 0x7f561006e090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56300cd880 0x7f5630154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56300cb6b0 0x7f56301578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 --2- 172.21.15.92:0/3401081523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56300ca2e0 0x7f56301568d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 >> 172.21.15.92:0/3401081523 conn(0x7f5630072970 msgr2=0x7f56300cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 shutdown_connections 2024-03-19T02:55:12.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.336+0000 7f5636400640 1 -- 172.21.15.92:0/3401081523 wait complete. 2024-03-19T02:55:12.355 INFO:teuthology.run_tasks:Running task kclient... 2024-03-19T02:55:12.382 INFO:tasks.kclient:Mounting kernel clients... 2024-03-19T02:55:12.382 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-19T02:55:12.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:12.382 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-19T02:55:12.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:12.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link delete ceph-brx 2024-03-19T02:55:12.531 INFO:teuthology.orchestra.run.smithi092.stderr:Cannot find device "ceph-brx" 2024-03-19T02:55:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:55:12.535 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-19T02:55:12.535 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T02:55:12.535 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:12.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-19T02:55:12.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.742+0000 7f75fb200640 1 Processor -- start 2024-03-19T02:55:12.743 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.742+0000 7f75fb200640 1 -- start start 2024-03-19T02:55:12.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.743+0000 7f75fb200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75f4152bf0 0x7f75f406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.743+0000 7f75fb200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f75f40c6470 0x7f75f40c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.743+0000 7f75fb200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f40c9a80 con 0x7f75f41593d0 2024-03-19T02:55:12.745 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.743+0000 7f75fb200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40c6c90 con 0x7f75f4152bf0 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.743+0000 7f75fb200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f75f4159910 con 0x7f75f40c6470 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f75f40c6470 0x7f75f40c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75f4152bf0 0x7f75f406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f9200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f75f41593d0 0x7f75f40c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52436/0 (socket says 172.21.15.92:52436) 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f75f40c6470 0x7f75f40c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36870/0 (socket says 172.21.15.92:36870) 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f9200640 1 -- 172.21.15.92:0/483449378 learned_addr learned my addr 172.21.15.92:0/483449378 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 -- 172.21.15.92:0/483449378 >> v2:172.21.15.92:3300/0 conn(0x7f75f4152bf0 msgr2=0x7f75f406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 --2- 172.21.15.92:0/483449378 >> v2:172.21.15.92:3300/0 conn(0x7f75f4152bf0 0x7f75f406eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 -- 172.21.15.92:0/483449378 >> v1:172.21.15.159:6789/0 conn(0x7f75f41593d0 legacy=0x7f75f40c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f3400640 1 -- 172.21.15.92:0/483449378 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40c8650 con 0x7f75f40c6470 2024-03-19T02:55:12.746 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.744+0000 7f75f2a00640 1 -- 172.21.15.92:0/483449378 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246003050 0 0) 0x7f75f40c9a80 con 0x7f75f41593d0 2024-03-19T02:55:12.747 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.745+0000 7f75f3400640 1 --2- 172.21.15.92:0/483449378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f40c3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75f4075340 tx=0x7f75e002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba8083f82cb11bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75f2a00640 1 -- 172.21.15.92:0/483449378 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e00315a0 con 0x7f75f40c6470 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75f2a00640 1 -- 172.21.15.92:0/483449378 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e0037070 con 0x7f75f40c6470 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75f2a00640 1 -- 172.21.15.92:0/483449378 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031960 con 0x7f75f40c6470 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 -- 172.21.15.92:0/483449378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 msgr2=0x7f75f40c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 --2- 172.21.15.92:0/483449378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f40c3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75f4075340 tx=0x7f75e002b840 comp rx=0 tx=0).stop 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 -- 172.21.15.92:0/483449378 shutdown_connections 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 --2- 172.21.15.92:0/483449378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f40c3f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 --2- 172.21.15.92:0/483449378 >> v2:172.21.15.92:3300/0 conn(0x7f75f4152bf0 0x7f75f406eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 -- 172.21.15.92:0/483449378 >> 172.21.15.92:0/483449378 conn(0x7f75f40c8d90 msgr2=0x7f75f40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 -- 172.21.15.92:0/483449378 shutdown_connections 2024-03-19T02:55:12.748 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.746+0000 7f75fb200640 1 -- 172.21.15.92:0/483449378 wait complete. 2024-03-19T02:55:12.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.748+0000 7f75fb200640 1 Processor -- start 2024-03-19T02:55:12.749 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.748+0000 7f75fb200640 1 -- start start 2024-03-19T02:55:12.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.751 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75f4152bf0 0x7f75f415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f41593d0 0x7f75f415b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75f406ea60 con 0x7f75f41593d0 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f75f406e4a0 con 0x7f75f40c6470 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75fb200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f75f406e6f0 con 0x7f75f4152bf0 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75f3e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75f3e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:36882/0 (socket says 172.21.15.92:36882) 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75f3400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75f4152bf0 0x7f75f415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75f3e00640 1 -- 172.21.15.92:0/2589455584 learned_addr learned my addr 172.21.15.92:0/2589455584 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.749+0000 7f75f9200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f41593d0 0x7f75f415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 -- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75f4152bf0 msgr2=0x7f75f415b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75f4152bf0 0x7f75f415b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 -- 172.21.15.92:0/2589455584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f41593d0 msgr2=0x7f75f415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f41593d0 0x7f75f415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 -- 172.21.15.92:0/2589455584 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41f0a10 con 0x7f75f40c6470 2024-03-19T02:55:12.752 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75f3e00640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75e400e9e0 tx=0x7f75e400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4017070 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e400c9d0 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e400cdc0 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f414c4a0 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.750+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41f0f70 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.752+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f40684c0 con 0x7f75f40c6470 2024-03-19T02:55:12.753 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.752+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f75e401a070 con 0x7f75f40c6470 2024-03-19T02:55:12.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.753+0000 7f75ebe00640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:12.754 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.753+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f75e408aae0 con 0x7f75f40c6470 2024-03-19T02:55:12.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.754+0000 7f75f3400640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:12.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.754+0000 7f75f3400640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 0x7f75d406e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75e0036000 tx=0x7f75e003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:12.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.758+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f75f41f0a10 con 0x7f75f40c6470 2024-03-19T02:55:12.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.993+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75f406dff0 con 0x7f75f40c6470 2024-03-19T02:55:12.995 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.994+0000 7f75ebe00640 1 -- 172.21.15.92:0/2589455584 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f75e4058ef0 con 0x7f75f40c6470 2024-03-19T02:55:12.995 INFO:teuthology.orchestra.run.smithi092.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-19T02:55:12.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 msgr2=0x7f75d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 0x7f75d406e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75e0036000 tx=0x7f75e003d040 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 msgr2=0x7f75f415c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75e400e9e0 tx=0x7f75e400eea0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 shutdown_connections 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f41593d0 0x7f75f415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75f4152bf0 0x7f75f415b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 --2- 172.21.15.92:0/2589455584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75f40c6470 0x7f75f415c810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 >> 172.21.15.92:0/2589455584 conn(0x7f75f40c8d90 msgr2=0x7f75f40c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 shutdown_connections 2024-03-19T02:55:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:12.996+0000 7f75fb200640 1 -- 172.21.15.92:0/2589455584 wait complete. 2024-03-19T02:55:13.017 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T02:55:13.017 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi092.front.sepia.ceph.com 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T02:55:13.018 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:55:13.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:13.018 DEBUG:teuthology.orchestra.run.smithi092:> ip addr 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b2 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f0 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:55:13.068 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b3 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f1 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:44 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:45 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: inet 172.21.15.92/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft 41983sec preferred_lft 41983sec 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-03-19T02:55:13.069 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:55:13.071 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link add name ceph-brx type bridge 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip addr flush dev ceph-brx 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set ceph-brx up 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T02:55:13.071 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-19T02:55:13.174 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:13.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:13.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:13.579 DEBUG:teuthology.orchestra.run.smithi092:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T02:55:13.653 INFO:teuthology.orchestra.run.smithi092.stdout:1 2024-03-19T02:55:13.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:13.658 DEBUG:teuthology.orchestra.run.smithi092:> route 2024-03-19T02:55:13.723 INFO:teuthology.orchestra.run.smithi092.stdout:Kernel IP routing table 2024-03-19T02:55:13.723 INFO:teuthology.orchestra.run.smithi092.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T02:55:13.724 INFO:teuthology.orchestra.run.smithi092.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T02:55:13.724 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T02:55:13.724 INFO:teuthology.orchestra.run.smithi092.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T02:55:13.724 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:55:13.724 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-19T02:55:13.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T02:55:13.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T02:55:13.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T02:55:13.725 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-19T02:55:13.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:14.539 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:14.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:14.549 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-19T02:55:14.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:14.601 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list-id 2024-03-19T02:55:14.651 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:55:14.651 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-19T02:55:14.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:55:14.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T02:55:14.652 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-19T02:55:14.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:14.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:14.858 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T02:55:14.859 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-19T02:55:14.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:16.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:16.140 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:55:16.140 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-19T02:55:16.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set brx.0 up 2024-03-19T02:55:16.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T02:55:16.140 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-19T02:55:16.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:16.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:16.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:55:16.365 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:16.412 INFO:teuthology.orchestra.run.smithi092.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:55:16.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:55:16.413 DEBUG:teuthology.orchestra.run.smithi092:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:16.462 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-19T02:55:16.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:16.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T02:55:16.601 INFO:teuthology.orchestra.run.smithi092.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T02:55:16.602 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T02:55:16.602 INFO:teuthology.orchestra.run.smithi092.stdout:invalid new device string format 2024-03-19T02:55:17.228 INFO:teuthology.orchestra.run.smithi092.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T02:55:17.228 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T02:55:17.228 INFO:teuthology.orchestra.run.smithi092.stdout:invalid new device string format 2024-03-19T02:55:17.228 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: resolved to: "172.21.15.92:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-19T02:55:17.228 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:invalid new device string format 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: resolved to: "172.21.15.92:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: resolved to: "172.21.15.92:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.92:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-03-19T02:55:17.264 INFO:teuthology.orchestra.run.smithi092.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=91932042-1f9e-4210-9e82-70c11e6277fd" will pass to kernel 2024-03-19T02:55:17.269 INFO:teuthology.orchestra.run.smithi092.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-19T02:55:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: You just mounted a file system that supports labels which does not 2024-03-19T02:55:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-19T02:55:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: applications will generate AVC messages and not be allowed access to 2024-03-19T02:55:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-19T02:55:17.275 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-19T02:55:17.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:17.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:17.363 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T02:55:17.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T02:55:17.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:17.486 INFO:teuthology.orchestra.run.smithi092.stdout:client4539 2024-03-19T02:55:17.486 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:55:17.486 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:55:17.537 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T02:55:17.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T02:55:17.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:17.712 INFO:teuthology.orchestra.run.smithi092.stdout:client4539 2024-03-19T02:55:17.712 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T02:55:17.820 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T02:55:17.830 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:17.839 INFO:teuthology.orchestra.run.smithi092.stdout:client4539 2024-03-19T02:55:17.840 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/91932042-1f9e-4210-9e82-70c11e6277fd.client4539/status') 2024-03-19T02:55:18.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:55:18.985 INFO:teuthology.orchestra.run.smithi092.stdout:instance: client.4539 (3)192.168.0.1:0/4100718164 2024-03-19T02:55:18.985 INFO:teuthology.orchestra.run.smithi092.stdout:blocklisted: false 2024-03-19T02:55:18.985 INFO:teuthology.orchestra.run.smithi092.stderr:0+1 records in 2024-03-19T02:55:18.985 INFO:teuthology.orchestra.run.smithi092.stderr:0+1 records out 2024-03-19T02:55:18.985 INFO:teuthology.orchestra.run.smithi092.stderr:69 bytes copied, 7.4405e-05 s, 927 kB/s 2024-03-19T02:55:18.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:55:18.999 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-03-19T02:55:19.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:19.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.225+0000 7f6d0e600640 1 Processor -- start 2024-03-19T02:55:19.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.225+0000 7f6d0e600640 1 -- start start 2024-03-19T02:55:19.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6d080c6470 0x7f6d080c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d081593d0 0x7f6d080c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6d080c6c90 con 0x7f6d081593d0 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6d08159910 con 0x7f6d080c6470 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.226+0000 7f6d0e600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0806e4a0 con 0x7f6d08152bf0 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6cffe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6d080c6470 0x7f6d080c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41588/0 (socket says 172.21.15.92:41588) 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 -- 172.21.15.92:0/2133511716 learned_addr learned my addr 172.21.15.92:0/2133511716 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d081593d0 0x7f6d080c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 -- 172.21.15.92:0/2133511716 >> v2:172.21.15.159:3300/0 conn(0x7f6d080c6470 msgr2=0x7f6d080c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 --2- 172.21.15.92:0/2133511716 >> v2:172.21.15.159:3300/0 conn(0x7f6d080c6470 0x7f6d080c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 -- 172.21.15.92:0/2133511716 >> v2:172.21.15.92:3300/0 conn(0x7f6d081593d0 msgr2=0x7f6d080c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 --2- 172.21.15.92:0/2133511716 >> v2:172.21.15.92:3300/0 conn(0x7f6d081593d0 0x7f6d080c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.227+0000 7f6d07400640 1 -- 172.21.15.92:0/2133511716 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d080c9a80 con 0x7f6d08152bf0 2024-03-19T02:55:19.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.228+0000 7f6d07400640 1 --2- 172.21.15.92:0/2133511716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d08075310 tx=0x7f6cf802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc415b6f5cb2519e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d06a00640 1 -- 172.21.15.92:0/2133511716 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf802e070 con 0x7f6d08152bf0 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d06a00640 1 -- 172.21.15.92:0/2133511716 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf8037070 con 0x7f6d08152bf0 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d06a00640 1 -- 172.21.15.92:0/2133511716 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8031a00 con 0x7f6d08152bf0 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 -- 172.21.15.92:0/2133511716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 msgr2=0x7f6d0806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 --2- 172.21.15.92:0/2133511716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d08075310 tx=0x7f6cf802b880 comp rx=0 tx=0).stop 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 -- 172.21.15.92:0/2133511716 shutdown_connections 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 --2- 172.21.15.92:0/2133511716 >> v2:172.21.15.92:3300/0 conn(0x7f6d081593d0 0x7f6d080c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 --2- 172.21.15.92:0/2133511716 >> v2:172.21.15.159:3300/0 conn(0x7f6d080c6470 0x7f6d080c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 --2- 172.21.15.92:0/2133511716 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d0806eef0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 -- 172.21.15.92:0/2133511716 >> 172.21.15.92:0/2133511716 conn(0x7f6d080c8d90 msgr2=0x7f6d080c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 -- 172.21.15.92:0/2133511716 shutdown_connections 2024-03-19T02:55:19.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.229+0000 7f6d0e600640 1 -- 172.21.15.92:0/2133511716 wait complete. 2024-03-19T02:55:19.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.231+0000 7f6d0e600640 1 Processor -- start 2024-03-19T02:55:19.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.232+0000 7f6d0e600640 1 -- start start 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 0x7f6d0815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d081593d0 0x7f6d0815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6d080c8930 con 0x7f6d081593d0 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0806ea60 con 0x7f6d080c6470 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d0e600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0806e6f0 con 0x7f6d08152bf0 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d07400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 0x7f6d0815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41600/0 (socket says 172.21.15.92:41600) 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 -- 172.21.15.92:0/88962290 learned_addr learned my addr 172.21.15.92:0/88962290 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d07400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 0x7f6d0815a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:45992/0 (socket says 172.21.15.92:45992) 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6d07e00640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d081593d0 0x7f6d0815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 -- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 msgr2=0x7f6d0815a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 0x7f6d0815a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 -- 172.21.15.92:0/88962290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d081593d0 msgr2=0x7f6d0815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d081593d0 0x7f6d0815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.233+0000 7f6cffe00640 1 -- 172.21.15.92:0/88962290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d081f0cf0 con 0x7f6d08152bf0 2024-03-19T02:55:19.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6cffe00640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e890 tx=0x7f6cf000ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0017070 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf000c940 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000cd10 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d08071230 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.234+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d080717f0 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.235+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d080684c0 con 0x7f6d08152bf0 2024-03-19T02:55:19.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.236+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6cf001a070 con 0x7f6d08152bf0 2024-03-19T02:55:19.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.237+0000 7f6cff400640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 0x7f6ce406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.237+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6cf008acf0 con 0x7f6d08152bf0 2024-03-19T02:55:19.237 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.237+0000 7f6d07400640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 0x7f6ce406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.238+0000 7f6d07400640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 0x7f6ce406e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.241+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6cf0059180 con 0x7f6d08152bf0 2024-03-19T02:55:19.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.441+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d08148e50 con 0x7f6d08152bf0 2024-03-19T02:55:19.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.443+0000 7f6cff400640 1 -- 172.21.15.92:0/88962290 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f6cf005d220 con 0x7f6d08152bf0 2024-03-19T02:55:19.443 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:19.444 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.445+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 msgr2=0x7f6ce406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.445+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 0x7f6ce406e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803d040 comp rx=0 tx=0).stop 2024-03-19T02:55:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.445+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 msgr2=0x7f6d081e62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.445+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e890 tx=0x7f6cf000ed50 comp rx=0 tx=0).stop 2024-03-19T02:55:19.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 shutdown_connections 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ce406bbd0 0x7f6ce406e090 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d081593d0 0x7f6d0815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d08152bf0 0x7f6d081e62a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 --2- 172.21.15.92:0/88962290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d080c6470 0x7f6d0815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 >> 172.21.15.92:0/88962290 conn(0x7f6d080c8d90 msgr2=0x7f6d080c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 shutdown_connections 2024-03-19T02:55:19.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.446+0000 7f6d0e600640 1 -- 172.21.15.92:0/88962290 wait complete. 2024-03-19T02:55:19.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:55:19.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:55:19.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.678+0000 7f1ad3400640 1 Processor -- start 2024-03-19T02:55:19.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.678+0000 7f1ad3400640 1 -- start start 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.679+0000 7f1ad3400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1acc1593d0 0x7f1acc0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.679+0000 7f1ad3400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1acc0c9a80 con 0x7f1acc152bf0 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad3400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1acc0c8650 con 0x7f1acc0c6470 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad3400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1acc0c6c90 con 0x7f1acc1593d0 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad1400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1acc1593d0 0x7f1acc0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad0a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1acc152bf0 0x7f1acc06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46174/0 (socket says 172.21.15.92:46174) 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1acbe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1acc0c6470 0x7f1acc0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49336/0 (socket says 172.21.15.92:49336) 2024-03-19T02:55:19.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad0a00640 1 -- 172.21.15.92:0/1048260932 learned_addr learned my addr 172.21.15.92:0/1048260932 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:19.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad1400640 1 -- 172.21.15.92:0/1048260932 >> v1:172.21.15.159:6789/0 conn(0x7f1acc0c6470 legacy=0x7f1acc0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad1400640 1 -- 172.21.15.92:0/1048260932 >> v1:172.21.15.92:6789/0 conn(0x7f1acc152bf0 legacy=0x7f1acc06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1acb400640 1 -- 172.21.15.92:0/1048260932 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699386945 0 0) 0x7f1acc0c9a80 con 0x7f1acc152bf0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.680+0000 7f1ad1400640 1 -- 172.21.15.92:0/1048260932 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc0c8930 con 0x7f1acc1593d0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.681+0000 7f1ad1400640 1 --2- 172.21.15.92:0/1048260932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc0c4690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009d90 tx=0x7f1ac402bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f598d5ae371fec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1acb400640 1 -- 172.21.15.92:0/1048260932 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4039070 con 0x7f1acc1593d0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1acb400640 1 -- 172.21.15.92:0/1048260932 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac402ca50 con 0x7f1acc1593d0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1acb400640 1 -- 172.21.15.92:0/1048260932 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac402ce20 con 0x7f1acc1593d0 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 -- 172.21.15.92:0/1048260932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 msgr2=0x7f1acc0c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 --2- 172.21.15.92:0/1048260932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc0c4690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009d90 tx=0x7f1ac402bd20 comp rx=0 tx=0).stop 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 -- 172.21.15.92:0/1048260932 shutdown_connections 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 --2- 172.21.15.92:0/1048260932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc0c4690 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 -- 172.21.15.92:0/1048260932 >> 172.21.15.92:0/1048260932 conn(0x7f1acc0c8d90 msgr2=0x7f1acc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 -- 172.21.15.92:0/1048260932 shutdown_connections 2024-03-19T02:55:19.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.682+0000 7f1ad3400640 1 -- 172.21.15.92:0/1048260932 wait complete. 2024-03-19T02:55:19.684 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.684+0000 7f1ad3400640 1 Processor -- start 2024-03-19T02:55:19.684 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.684+0000 7f1ad3400640 1 -- start start 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1acc0c6470 0x7f1acc0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1acc152bf0 0x7f1acc1f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc06e4a0 con 0x7f1acc0c6470 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc159910 con 0x7f1acc1593d0 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad3400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1acc06ea60 con 0x7f1acc152bf0 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad0a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1acc0c6470 0x7f1acc0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad1400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1acbe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1acc152bf0 0x7f1acc1f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad1400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46008/0 (socket says 172.21.15.92:46008) 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.685+0000 7f1ad1400640 1 -- 172.21.15.92:0/3152243837 learned_addr learned my addr 172.21.15.92:0/3152243837 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:19.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 -- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1acc152bf0 msgr2=0x7f1acc1f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1acc152bf0 0x7f1acc1f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 -- 172.21.15.92:0/3152243837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1acc0c6470 msgr2=0x7f1acc0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1acc0c6470 0x7f1acc0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 -- 172.21.15.92:0/3152243837 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc0cb6a0 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad1400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009d90 tx=0x7f1ac4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4044030 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac402c850 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc146300 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4039040 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.686+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc1f1960 con 0x7f1acc1593d0 2024-03-19T02:55:19.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.687+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1acc0684c0 con 0x7f1acc1593d0 2024-03-19T02:55:19.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.688+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ac4047070 con 0x7f1acc1593d0 2024-03-19T02:55:19.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.689+0000 7f1ac8c00640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:19.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.689+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1ac40ad310 con 0x7f1acc1593d0 2024-03-19T02:55:19.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.689+0000 7f1ad0a00640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:19.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.690+0000 7f1ad0a00640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 0x7f1aac06e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0007f70 tx=0x7f1ac0007f00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:19.693 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.693+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ac404e070 con 0x7f1acc1593d0 2024-03-19T02:55:19.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.891+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1acc1565a0 con 0x7f1acc1593d0 2024-03-19T02:55:19.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.892+0000 7f1ac8c00640 1 -- 172.21.15.92:0/3152243837 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f1acc0cb6a0 con 0x7f1acc1593d0 2024-03-19T02:55:19.892 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:55:19.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 msgr2=0x7f1aac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 0x7f1aac06e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0007f70 tx=0x7f1ac0007f00 comp rx=0 tx=0).stop 2024-03-19T02:55:19.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 msgr2=0x7f1acc1e59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:19.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009d90 tx=0x7f1ac4004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 shutdown_connections 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1acc1593d0 0x7f1acc1e59b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1acc152bf0 0x7f1acc1f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 --2- 172.21.15.92:0/3152243837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1acc0c6470 0x7f1acc0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 >> 172.21.15.92:0/3152243837 conn(0x7f1acc0c8d90 msgr2=0x7f1acc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 shutdown_connections 2024-03-19T02:55:19.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:19.895+0000 7f1ad3400640 1 -- 172.21.15.92:0/3152243837 wait complete. 2024-03-19T02:55:19.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:55:20.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.125+0000 7f9b4ec00640 1 Processor -- start 2024-03-19T02:55:20.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.125+0000 7f9b4ec00640 1 -- start start 2024-03-19T02:55:20.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b4ec00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b480679f0 0x7f9b480c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b4ec00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4806d630 con 0x7f9b4815aca0 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b4ec00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4815b130 con 0x7f9b48155090 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b4ec00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f9b48155470 con 0x7f9b480679f0 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b4cc00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b480679f0 0x7f9b480c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b3f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9b4815aca0 0x7f9b480cb750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46180/0 (socket says 172.21.15.92:46180) 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.127+0000 7f9b3f400640 1 -- 172.21.15.92:0/2843494804 learned_addr learned my addr 172.21.15.92:0/2843494804 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.128+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2843494804 >> v1:172.21.15.159:6789/0 conn(0x7f9b48155090 legacy=0x7f9b4815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.128+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2843494804 >> v1:172.21.15.92:6789/0 conn(0x7f9b4815aca0 legacy=0x7f9b480cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.128+0000 7f9b47400640 1 -- 172.21.15.92:0/2843494804 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358528686 0 0) 0x7f9b30001110 con 0x7f9b4815aca0 2024-03-19T02:55:20.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.128+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2843494804 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4815b3f0 con 0x7f9b480679f0 2024-03-19T02:55:20.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.129+0000 7f9b4cc00640 1 --2- 172.21.15.92:0/2843494804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b480679f0 0x7f9b480c57f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b4002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6614c50ea2d912a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.129+0000 7f9b47400640 1 -- 172.21.15.92:0/2843494804 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40039070 con 0x7f9b480679f0 2024-03-19T02:55:20.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.129+0000 7f9b47400640 1 -- 172.21.15.92:0/2843494804 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4002cae0 con 0x7f9b480679f0 2024-03-19T02:55:20.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.129+0000 7f9b47400640 1 -- 172.21.15.92:0/2843494804 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40034440 con 0x7f9b480679f0 2024-03-19T02:55:20.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2843494804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b480679f0 msgr2=0x7f9b480c57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2843494804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b480679f0 0x7f9b480c57f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b4002bdb0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2843494804 shutdown_connections 2024-03-19T02:55:20.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2843494804 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b480679f0 0x7f9b480c57f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2843494804 >> 172.21.15.92:0/2843494804 conn(0x7f9b48072970 msgr2=0x7f9b48066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:20.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2843494804 shutdown_connections 2024-03-19T02:55:20.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.130+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2843494804 wait complete. 2024-03-19T02:55:20.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.132+0000 7f9b4ec00640 1 Processor -- start 2024-03-19T02:55:20.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.132+0000 7f9b4ec00640 1 -- start start 2024-03-19T02:55:20.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b480679f0 0x7f9b480b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4815e3f0 con 0x7f9b480679f0 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4815b9b0 con 0x7f9b48155090 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4ec00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4815f020 con 0x7f9b48153040 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b3f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b480679f0 0x7f9b480b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41610/0 (socket says 172.21.15.92:41610) 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b3f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46026/0 (socket says 172.21.15.92:46026) 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.133+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2773731307 learned_addr learned my addr 172.21.15.92:0/2773731307 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 msgr2=0x7f9b481535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2773731307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b480679f0 msgr2=0x7f9b480b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b480679f0 0x7f9b480b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 -- 172.21.15.92:0/2773731307 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4815cf50 con 0x7f9b48153040 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b3f400640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4cc00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b40004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40044030 con 0x7f9b48153040 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4002c8e0 con 0x7f9b48153040 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4815d590 con 0x7f9b48153040 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40039040 con 0x7f9b48153040 2024-03-19T02:55:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.134+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4815db50 con 0x7f9b48153040 2024-03-19T02:55:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.135+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b480768a0 con 0x7f9b48153040 2024-03-19T02:55:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.136+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b40047070 con 0x7f9b48153040 2024-03-19T02:55:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.137+0000 7f9b44c00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 0x7f9b2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.137+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9b400ad200 con 0x7f9b48153040 2024-03-19T02:55:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.137+0000 7f9b47e00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 0x7f9b2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.138+0000 7f9b47e00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 0x7f9b2406e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000f60 tx=0x7f9b38009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.141+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9b4004e070 con 0x7f9b48153040 2024-03-19T02:55:20.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.343+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b480cb530 con 0x7f9b48153040 2024-03-19T02:55:20.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.344+0000 7f9b44c00640 1 -- 172.21.15.92:0/2773731307 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f9b4002cab0 con 0x7f9b48153040 2024-03-19T02:55:20.345 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 msgr2=0x7f9b2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 0x7f9b2406e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b38000f60 tx=0x7f9b38009040 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 msgr2=0x7f9b48153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b40004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 shutdown_connections 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9b2406bbd0 0x7f9b2406e090 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b48153040 0x7f9b48153b10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b48155090 0x7f9b481535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 --2- 172.21.15.92:0/2773731307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b480679f0 0x7f9b480b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 >> 172.21.15.92:0/2773731307 conn(0x7f9b48072970 msgr2=0x7f9b480cb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 shutdown_connections 2024-03-19T02:55:20.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.347+0000 7f9b4ec00640 1 -- 172.21.15.92:0/2773731307 wait complete. 2024-03-19T02:55:20.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:55:20.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.581+0000 7f148ea00640 1 Processor -- start 2024-03-19T02:55:20.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.582+0000 7f148ea00640 1 -- start start 2024-03-19T02:55:20.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.583+0000 7f148ea00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f148806f800 0x7f14880c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.583+0000 7f148ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 0x7f1488159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.583+0000 7f148ea00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1488072890 con 0x7f1488071da0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f148ea00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f14880c6c90 con 0x7f148806f800 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f148ea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f14880c8200 con 0x7f1488158db0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f1487e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1488071da0 0x7f1488072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46188/0 (socket says 172.21.15.92:46188) 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f1487e00640 1 -- 172.21.15.92:0/771614166 learned_addr learned my addr 172.21.15.92:0/771614166 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f147fe00640 1 --2- 172.21.15.92:0/771614166 >> v2:172.21.15.159:3300/0 conn(0x7f148806f800 0x7f14880c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f148ca00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 0x7f1488159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f1487400640 1 -- 172.21.15.92:0/771614166 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041279243 0 0) 0x7f1488072890 con 0x7f1488071da0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f1487400640 1 -- 172.21.15.92:0/771614166 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f146c003520 con 0x7f1488071da0 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f147fe00640 1 -- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 msgr2=0x7f1488159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f147fe00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 0x7f1488159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f147fe00640 1 -- 172.21.15.92:0/771614166 >> v1:172.21.15.92:6789/0 conn(0x7f1488071da0 legacy=0x7f1488072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f147fe00640 1 -- 172.21.15.92:0/771614166 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14880c84e0 con 0x7f148806f800 2024-03-19T02:55:20.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.584+0000 7f148ca00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 0x7f1488159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.585+0000 7f147fe00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f148806f800 0x7f14880c8740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14700097a0 tx=0x7f147002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1a568785e495447 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f1487400640 1 -- 172.21.15.92:0/771614166 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470035020 con 0x7f148806f800 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f1487400640 1 -- 172.21.15.92:0/771614166 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1470031810 con 0x7f148806f800 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f1487400640 1 -- 172.21.15.92:0/771614166 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470031be0 con 0x7f148806f800 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 -- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f148806f800 msgr2=0x7f14880c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f148806f800 0x7f14880c8740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14700097a0 tx=0x7f147002b8b0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 -- 172.21.15.92:0/771614166 shutdown_connections 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1488158db0 0x7f1488159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 --2- 172.21.15.92:0/771614166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f148806f800 0x7f14880c8740 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.586+0000 7f148ea00640 1 -- 172.21.15.92:0/771614166 >> 172.21.15.92:0/771614166 conn(0x7f14880c8d90 msgr2=0x7f14880c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:20.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.587+0000 7f148ea00640 1 -- 172.21.15.92:0/771614166 shutdown_connections 2024-03-19T02:55:20.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.587+0000 7f148ea00640 1 -- 172.21.15.92:0/771614166 wait complete. 2024-03-19T02:55:20.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.588+0000 7f148ea00640 1 Processor -- start 2024-03-19T02:55:20.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.588+0000 7f148ea00640 1 -- start start 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 0x7f14881f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 0x7f14880c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14880c7120 con 0x7f1488071da0 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f148815fde0 con 0x7f1488158db0 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f148ea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f14880c72a0 con 0x7f148806f800 2024-03-19T02:55:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f1487e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 0x7f14881f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.589+0000 7f147fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 0x7f14880c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46052/0 (socket says 172.21.15.92:46052) 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 -- 172.21.15.92:0/4088978013 learned_addr learned my addr 172.21.15.92:0/4088978013 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 -- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 msgr2=0x7f14881f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 0x7f14881f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 -- 172.21.15.92:0/4088978013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 msgr2=0x7f14880c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 0x7f14880c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 -- 172.21.15.92:0/4088978013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14881630a0 con 0x7f1488158db0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f147fe00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 0x7f14880c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f1487e00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 0x7f14881f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f148ca00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f148000efc0 tx=0x7f148000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480018030 con 0x7f1488158db0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.590+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148000cdf0 con 0x7f1488158db0 2024-03-19T02:55:20.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.591+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480003bb0 con 0x7f1488158db0 2024-03-19T02:55:20.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.591+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1488162d60 con 0x7f1488158db0 2024-03-19T02:55:20.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.591+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14881f4a80 con 0x7f1488158db0 2024-03-19T02:55:20.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.592+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1488068de0 con 0x7f1488158db0 2024-03-19T02:55:20.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.593+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1480005d10 con 0x7f1488158db0 2024-03-19T02:55:20.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.593+0000 7f1484c00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 0x7f146406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:20.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.594+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f148008b310 con 0x7f1488158db0 2024-03-19T02:55:20.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.594+0000 7f1487e00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 0x7f146406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:20.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.594+0000 7f1487e00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 0x7f146406e0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1478005880 tx=0x7f1478009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:20.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.598+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1488068de0 con 0x7f1488158db0 2024-03-19T02:55:20.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.795+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f14880684c0 con 0x7f1488158db0 2024-03-19T02:55:20.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.796+0000 7f1484c00640 1 -- 172.21.15.92:0/4088978013 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f1480022070 con 0x7f1488158db0 2024-03-19T02:55:20.796 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T02:55:20.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 msgr2=0x7f146406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 0x7f146406e0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1478005880 tx=0x7f1478009040 comp rx=0 tx=0).stop 2024-03-19T02:55:20.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 msgr2=0x7f1488153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f148000efc0 tx=0x7f148000c490 comp rx=0 tx=0).stop 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 shutdown_connections 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f146406bc00 0x7f146406e0c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1488158db0 0x7f1488153a90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1488071da0 0x7f14880c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 --2- 172.21.15.92:0/4088978013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f148806f800 0x7f14881f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 >> 172.21.15.92:0/4088978013 conn(0x7f14880c8d90 msgr2=0x7f1488157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 shutdown_connections 2024-03-19T02:55:20.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:20.799+0000 7f148ea00640 1 -- 172.21.15.92:0/4088978013 wait complete. 2024-03-19T02:55:20.818 INFO:tasks.mds_thrash:mds thrasher using random seed: 1710816920 2024-03-19T02:55:20.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:21.019 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.019+0000 7f8195a00640 1 Processor -- start 2024-03-19T02:55:21.020 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.019+0000 7f8195a00640 1 -- start start 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8195a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8195a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 0x7f81900c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8195a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81900c9a80 con 0x7f81900c6470 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8195a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f81900c6c90 con 0x7f8190152bf0 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8195a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8190159910 con 0x7f81901593d0 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818ea00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 0x7f81900c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818ea00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46066/0 (socket says 172.21.15.92:46066) 2024-03-19T02:55:21.022 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818ea00640 1 -- 172.21.15.92:0/645994511 learned_addr learned my addr 172.21.15.92:0/645994511 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f8186000640 1 --1- 172.21.15.92:0/645994511 >> v1:172.21.15.92:6789/0 conn(0x7f81900c6470 0x7f81900c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46198/0 (socket says 172.21.15.92:46198) 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818f400640 1 -- 172.21.15.92:0/645994511 >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 msgr2=0x7f819006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818f400640 1 --2- 172.21.15.92:0/645994511 >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818f400640 1 -- 172.21.15.92:0/645994511 >> v1:172.21.15.92:6789/0 conn(0x7f81900c6470 legacy=0x7f81900c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818f400640 1 -- 172.21.15.92:0/645994511 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81900c8650 con 0x7f81901593d0 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818ea00640 1 --2- 172.21.15.92:0/645994511 >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:21.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.021+0000 7f818e000640 1 -- 172.21.15.92:0/645994511 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572430233 0 0) 0x7f81900c9a80 con 0x7f81900c6470 2024-03-19T02:55:21.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f818f400640 1 --2- 172.21.15.92:0/645994511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 0x7f81900c4690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8188009d90 tx=0x7f818802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2da89f847c5f9bef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f818e000640 1 -- 172.21.15.92:0/645994511 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188039070 con 0x7f81901593d0 2024-03-19T02:55:21.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f818e000640 1 -- 172.21.15.92:0/645994511 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818802cbd0 con 0x7f81901593d0 2024-03-19T02:55:21.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f818e000640 1 -- 172.21.15.92:0/645994511 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188034440 con 0x7f81901593d0 2024-03-19T02:55:21.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f8195a00640 1 -- 172.21.15.92:0/645994511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 msgr2=0x7f81900c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.023+0000 7f8195a00640 1 --2- 172.21.15.92:0/645994511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 0x7f81900c4690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8188009d90 tx=0x7f818802be20 comp rx=0 tx=0).stop 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 -- 172.21.15.92:0/645994511 shutdown_connections 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 --2- 172.21.15.92:0/645994511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81901593d0 0x7f81900c4690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 --2- 172.21.15.92:0/645994511 >> v2:172.21.15.159:3300/0 conn(0x7f8190152bf0 0x7f819006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 -- 172.21.15.92:0/645994511 >> 172.21.15.92:0/645994511 conn(0x7f81900c8d90 msgr2=0x7f81900c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 -- 172.21.15.92:0/645994511 shutdown_connections 2024-03-19T02:55:21.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.024+0000 7f8195a00640 1 -- 172.21.15.92:0/645994511 wait complete. 2024-03-19T02:55:21.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.026+0000 7f8195a00640 1 Processor -- start 2024-03-19T02:55:21.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.026+0000 7f8195a00640 1 -- start start 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81900c6470 0x7f819014fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8190152bf0 0x7f81901e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f819006ea60 con 0x7f81900c6470 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f819006e4a0 con 0x7f81901593d0 2024-03-19T02:55:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.027+0000 7f8195a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f819006e6f0 con 0x7f8190152bf0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f8186000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8190152bf0 0x7f81901e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46070/0 (socket says 172.21.15.92:46070) 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 -- 172.21.15.92:0/2285619926 learned_addr learned my addr 172.21.15.92:0/2285619926 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818ea00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81900c6470 0x7f819014fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 -- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8190152bf0 msgr2=0x7f81901e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8190152bf0 0x7f81901e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 -- 172.21.15.92:0/2285619926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81900c6470 msgr2=0x7f819014fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81900c6470 0x7f819014fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 -- 172.21.15.92:0/2285619926 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81901f47c0 con 0x7f81901593d0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f818f400640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8188009d90 tx=0x7f8188004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.028+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188039050 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.029+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188045070 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.029+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188009450 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.029+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8190150050 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.029+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81901f4d40 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.030+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8190068de0 con 0x7f81901593d0 2024-03-19T02:55:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.031+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f818803e850 con 0x7f81901593d0 2024-03-19T02:55:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.032+0000 7f8187400640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 0x7f816c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.032+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f81880ad9b0 con 0x7f81901593d0 2024-03-19T02:55:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.032+0000 7f818ea00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 0x7f816c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.033 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.033+0000 7f818ea00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 0x7f816c06e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8180009b70 tx=0x7f8180008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.036+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8190068de0 con 0x7f81901593d0 2024-03-19T02:55:21.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.234+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f819006f3d0 con 0x7f81901593d0 2024-03-19T02:55:21.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.235+0000 7f8187400640 1 -- 172.21.15.92:0/2285619926 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f818807c490 con 0x7f81901593d0 2024-03-19T02:55:21.236 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:21.236 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 msgr2=0x7f816c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 0x7f816c06e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8180009b70 tx=0x7f8180008040 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 msgr2=0x7f81901f1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8188009d90 tx=0x7f8188004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 shutdown_connections 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f816c06bbd0 0x7f816c06e090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81901593d0 0x7f81901f1250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8190152bf0 0x7f81901e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 --2- 172.21.15.92:0/2285619926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81900c6470 0x7f819014fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.238+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 >> 172.21.15.92:0/2285619926 conn(0x7f81900c8d90 msgr2=0x7f8190157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.239+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 shutdown_connections 2024-03-19T02:55:21.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.239+0000 7f8195a00640 1 -- 172.21.15.92:0/2285619926 wait complete. 2024-03-19T02:55:21.257 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:55:21.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:55:21.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.456+0000 7fcf61600640 1 Processor -- start 2024-03-19T02:55:21.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.456+0000 7fcf61600640 1 -- start start 2024-03-19T02:55:21.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.457+0000 7fcf61600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcf5c151900 0x7fcf5c158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.457+0000 7fcf61600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.457+0000 7fcf61600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c0c3b80 con 0x7fcf5c151ce0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.457+0000 7fcf61600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcf5c158860 con 0x7fcf5c151900 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.457+0000 7fcf61600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c072e70 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5a000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fcf5c151ce0 0x7fcf5c0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49360/0 (socket says 172.21.15.92:49360) 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5a000640 1 -- 172.21.15.92:0/3003500098 learned_addr learned my addr 172.21.15.92:0/3003500098 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/3003500098 >> v2:172.21.15.92:3300/0 conn(0x7fcf5c151900 0x7fcf5c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5b400640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf59600640 1 -- 172.21.15.92:0/3003500098 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258080628 0 0) 0x7fcf5c0c3b80 con 0x7fcf5c151ce0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf59600640 1 -- 172.21.15.92:0/3003500098 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf40003520 con 0x7fcf5c151ce0 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5aa00640 1 -- 172.21.15.92:0/3003500098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 msgr2=0x7fcf5c0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5aa00640 1 -- 172.21.15.92:0/3003500098 >> v1:172.21.15.159:6789/0 conn(0x7fcf5c151ce0 legacy=0x7fcf5c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5aa00640 1 -- 172.21.15.92:0/3003500098 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c073150 con 0x7fcf5c151900 2024-03-19T02:55:21.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.458+0000 7fcf5b400640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.459+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151900 0x7fcf5c158320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0097a0 tx=0x7fcf4c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fce9addeb15377 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.459+0000 7fcf59600640 1 -- 172.21.15.92:0/3003500098 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c034070 con 0x7fcf5c151900 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf59600640 1 -- 172.21.15.92:0/3003500098 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4c030800 con 0x7fcf5c151900 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf59600640 1 -- 172.21.15.92:0/3003500098 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c030bd0 con 0x7fcf5c151900 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 -- 172.21.15.92:0/3003500098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151900 msgr2=0x7fcf5c158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151900 0x7fcf5c158320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0097a0 tx=0x7fcf4c02b940 comp rx=0 tx=0).stop 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 -- 172.21.15.92:0/3003500098 shutdown_connections 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 --2- 172.21.15.92:0/3003500098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151900 0x7fcf5c158320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 -- 172.21.15.92:0/3003500098 >> 172.21.15.92:0/3003500098 conn(0x7fcf5c0c8d90 msgr2=0x7fcf5c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:21.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 -- 172.21.15.92:0/3003500098 shutdown_connections 2024-03-19T02:55:21.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.460+0000 7fcf61600640 1 -- 172.21.15.92:0/3003500098 wait complete. 2024-03-19T02:55:21.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.462+0000 7fcf61600640 1 Processor -- start 2024-03-19T02:55:21.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.462+0000 7fcf61600640 1 -- start start 2024-03-19T02:55:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf5c151900 0x7fcf5c1f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 0x7fcf5c1ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c0c9af0 con 0x7fcf5c151ce0 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c071eb0 con 0x7fcf5c151900 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf61600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c0c9c70 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf5b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 0x7fcf5c1ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf5aa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf5a000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf5c151900 0x7fcf5c1f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf5aa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41646/0 (socket says 172.21.15.92:41646) 2024-03-19T02:55:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.463+0000 7fcf5aa00640 1 -- 172.21.15.92:0/1564381589 learned_addr learned my addr 172.21.15.92:0/1564381589 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 -- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf5c151900 msgr2=0x7fcf5c1f06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf5c151900 0x7fcf5c1f06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 -- 172.21.15.92:0/1564381589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 msgr2=0x7fcf5c1ea2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 0x7fcf5c1ea2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 -- 172.21.15.92:0/1564381589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c0cad50 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5b400640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 0x7fcf5c1ea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf5aa00640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02bf50 tx=0x7fcf4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c048070 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4c030e30 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c1f3d90 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c034040 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.464+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c1f4350 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.465+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5c0684c0 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.466+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcf4c04b070 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.467+0000 7fcf52a00640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 0x7fcf3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.467+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcf4c0ac4d0 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.467+0000 7fcf5a000640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 0x7fcf3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.468+0000 7fcf5a000640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 0x7fcf3c06e090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c157530 tx=0x7fcf48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.471+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcf4c07a960 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.669+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcf5c072680 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.669+0000 7fcf52a00640 1 -- 172.21.15.92:0/1564381589 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fcf4c07ea00 con 0x7fcf5c0c7da0 2024-03-19T02:55:21.670 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:55:21.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 msgr2=0x7fcf3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 0x7fcf3c06e090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c157530 tx=0x7fcf48009040 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 msgr2=0x7fcf5c0c3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02bf50 tx=0x7fcf4c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 shutdown_connections 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcf3c06bbd0 0x7fcf3c06e090 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf5c151ce0 0x7fcf5c1ea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf5c151900 0x7fcf5c1f06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 --2- 172.21.15.92:0/1564381589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf5c0c7da0 0x7fcf5c0c3250 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 >> 172.21.15.92:0/1564381589 conn(0x7fcf5c0c8d90 msgr2=0x7fcf5c0c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 shutdown_connections 2024-03-19T02:55:21.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.672+0000 7fcf61600640 1 -- 172.21.15.92:0/1564381589 wait complete. 2024-03-19T02:55:21.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:55:21.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.896+0000 7f9495200640 1 Processor -- start 2024-03-19T02:55:21.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.896+0000 7f9495200640 1 -- start start 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f9495200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f94900c6470 0x7f94900c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f9495200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94900c9a80 con 0x7f9490152bf0 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f9495200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94900c8650 con 0x7f94901593d0 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f9495200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f94900c6c90 con 0x7f94900c6470 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948e800640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f94901593d0 0x7f94900c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49362/0 (socket says 172.21.15.92:49362) 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948e800640 1 -- 172.21.15.92:0/3896361021 learned_addr learned my addr 172.21.15.92:0/3896361021 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:21.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948d400640 1 --2- 172.21.15.92:0/3896361021 >> v2:172.21.15.159:3300/0 conn(0x7f94900c6470 0x7f94900c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948ca00640 1 -- 172.21.15.92:0/3896361021 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766401737 0 0) 0x7f94900c8650 con 0x7f94901593d0 2024-03-19T02:55:21.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948d400640 1 -- 172.21.15.92:0/3896361021 >> v1:172.21.15.159:6789/0 conn(0x7f94901593d0 legacy=0x7f94900c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948d400640 1 -- 172.21.15.92:0/3896361021 >> v1:172.21.15.92:6789/0 conn(0x7f9490152bf0 legacy=0x7f949006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.898+0000 7f948d400640 1 -- 172.21.15.92:0/3896361021 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94900c8930 con 0x7f94900c6470 2024-03-19T02:55:21.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.899+0000 7f948d400640 1 --2- 172.21.15.92:0/3896361021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94900c3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f947c0097a0 tx=0x7f947c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187a26b5951e6bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f948ca00640 1 -- 172.21.15.92:0/3896361021 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c035020 con 0x7f94900c6470 2024-03-19T02:55:21.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f948ca00640 1 -- 172.21.15.92:0/3896361021 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f947c031830 con 0x7f94900c6470 2024-03-19T02:55:21.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f948ca00640 1 -- 172.21.15.92:0/3896361021 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c031c00 con 0x7f94900c6470 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 -- 172.21.15.92:0/3896361021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 msgr2=0x7f94900c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 --2- 172.21.15.92:0/3896361021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94900c3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f947c0097a0 tx=0x7f947c02b8d0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 -- 172.21.15.92:0/3896361021 shutdown_connections 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 --2- 172.21.15.92:0/3896361021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94900c3f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 -- 172.21.15.92:0/3896361021 >> 172.21.15.92:0/3896361021 conn(0x7f94900c8d90 msgr2=0x7f94900c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.900+0000 7f9495200640 1 -- 172.21.15.92:0/3896361021 shutdown_connections 2024-03-19T02:55:21.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.901+0000 7f9495200640 1 -- 172.21.15.92:0/3896361021 wait complete. 2024-03-19T02:55:21.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.902+0000 7f9495200640 1 Processor -- start 2024-03-19T02:55:21.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.902+0000 7f9495200640 1 -- start start 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 0x7f94901f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94901593d0 0x7f94901f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f949006e4a0 con 0x7f9490152bf0 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9490159910 con 0x7f94900c6470 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.903+0000 7f9495200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f949006ea60 con 0x7f94901593d0 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 0x7f94901f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94901593d0 0x7f94901f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 0x7f94901f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49504/0 (socket says 172.21.15.92:49504) 2024-03-19T02:55:21.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948d400640 1 -- 172.21.15.92:0/1322916618 learned_addr learned my addr 172.21.15.92:0/1322916618 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:21.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46094/0 (socket says 172.21.15.92:46094) 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 -- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94901593d0 msgr2=0x7f94901f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94901593d0 0x7f94901f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 -- 172.21.15.92:0/1322916618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 msgr2=0x7f94901f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 0x7f94901f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 -- 172.21.15.92:0/1322916618 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490143880 con 0x7f94900c6470 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.904+0000 7f948de00640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f948000c390 tx=0x7f948000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.905+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480017070 con 0x7f94900c6470 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.905+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94800052d0 con 0x7f94900c6470 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.905+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9490143b40 con 0x7f94900c6470 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.905+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480003830 con 0x7f94900c6470 2024-03-19T02:55:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.905+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9490144100 con 0x7f94900c6470 2024-03-19T02:55:21.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.906+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94900684c0 con 0x7f94900c6470 2024-03-19T02:55:21.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.907+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f948001a070 con 0x7f94900c6470 2024-03-19T02:55:21.908 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.908+0000 7f9486000640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 0x7f947006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:21.908 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.908+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9480089da0 con 0x7f94900c6470 2024-03-19T02:55:21.908 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.908+0000 7f948d400640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 0x7f947006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:21.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.909+0000 7f948d400640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 0x7f947006e0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f94900c3470 tx=0x7f947c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:21.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:21.912+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f94800581a0 con 0x7f94900c6470 2024-03-19T02:55:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.109+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f949014cc70 con 0x7f94900c6470 2024-03-19T02:55:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.110+0000 7f9486000640 1 -- 172.21.15.92:0/1322916618 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f948005c240 con 0x7f94900c6470 2024-03-19T02:55:22.110 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T02:55:22.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.112+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 msgr2=0x7f947006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.112+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 0x7f947006e0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f94900c3470 tx=0x7f947c008030 comp rx=0 tx=0).stop 2024-03-19T02:55:22.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.112+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 msgr2=0x7f94901e5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.112+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f948000c390 tx=0x7f948000c850 comp rx=0 tx=0).stop 2024-03-19T02:55:22.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 shutdown_connections 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f947006bc00 0x7f947006e0c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f94901593d0 0x7f94901f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9490152bf0 0x7f94901f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 --2- 172.21.15.92:0/1322916618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f94900c6470 0x7f94901e5190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 >> 172.21.15.92:0/1322916618 conn(0x7f94900c8d90 msgr2=0x7f9490156720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 shutdown_connections 2024-03-19T02:55:22.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.113+0000 7f9495200640 1 -- 172.21.15.92:0/1322916618 wait complete. 2024-03-19T02:55:22.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:55:22.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.351+0000 7fca28e00640 1 Processor -- start 2024-03-19T02:55:22.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.351+0000 7fca28e00640 1 -- start start 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.352+0000 7fca28e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fca24152bf0 0x7fca2406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.352+0000 7fca28e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca240c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.352+0000 7fca28e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca240c9a80 con 0x7fca241593d0 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.352+0000 7fca28e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fca240c6c90 con 0x7fca24152bf0 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.352+0000 7fca28e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fca24159910 con 0x7fca240c6470 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca1be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca240c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca21e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fca24152bf0 0x7fca2406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca22800640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fca241593d0 0x7fca240c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49366/0 (socket says 172.21.15.92:49366) 2024-03-19T02:55:22.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca22800640 1 -- 172.21.15.92:0/3165941104 learned_addr learned my addr 172.21.15.92:0/3165941104 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:22.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca1be00640 1 -- 172.21.15.92:0/3165941104 >> v2:172.21.15.92:3300/0 conn(0x7fca24152bf0 msgr2=0x7fca2406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:22.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca1be00640 1 --2- 172.21.15.92:0/3165941104 >> v2:172.21.15.92:3300/0 conn(0x7fca24152bf0 0x7fca2406eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca21400640 1 -- 172.21.15.92:0/3165941104 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205161300 0 0) 0x7fca240c9a80 con 0x7fca241593d0 2024-03-19T02:55:22.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca1be00640 1 -- 172.21.15.92:0/3165941104 >> v1:172.21.15.159:6789/0 conn(0x7fca241593d0 legacy=0x7fca240c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.353+0000 7fca1be00640 1 -- 172.21.15.92:0/3165941104 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca240c8650 con 0x7fca240c6470 2024-03-19T02:55:22.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.354+0000 7fca1be00640 1 --2- 172.21.15.92:0/3165941104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca240c3f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0097a0 tx=0x7fca0c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7138a6eab7cbfbaa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.354+0000 7fca21400640 1 -- 172.21.15.92:0/3165941104 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c031780 con 0x7fca240c6470 2024-03-19T02:55:22.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.354+0000 7fca21400640 1 -- 172.21.15.92:0/3165941104 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c037070 con 0x7fca240c6470 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.354+0000 7fca21400640 1 -- 172.21.15.92:0/3165941104 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c031b40 con 0x7fca240c6470 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 -- 172.21.15.92:0/3165941104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 msgr2=0x7fca240c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 --2- 172.21.15.92:0/3165941104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca240c3f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0097a0 tx=0x7fca0c02b820 comp rx=0 tx=0).stop 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 -- 172.21.15.92:0/3165941104 shutdown_connections 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 --2- 172.21.15.92:0/3165941104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca240c3f80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 --2- 172.21.15.92:0/3165941104 >> v2:172.21.15.92:3300/0 conn(0x7fca24152bf0 0x7fca2406eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 -- 172.21.15.92:0/3165941104 >> 172.21.15.92:0/3165941104 conn(0x7fca240c8d90 msgr2=0x7fca240c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 -- 172.21.15.92:0/3165941104 shutdown_connections 2024-03-19T02:55:22.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.355+0000 7fca28e00640 1 -- 172.21.15.92:0/3165941104 wait complete. 2024-03-19T02:55:22.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.357+0000 7fca28e00640 1 Processor -- start 2024-03-19T02:55:22.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.357+0000 7fca28e00640 1 -- start start 2024-03-19T02:55:22.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca241593d0 0x7fca241f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fca2406ea60 con 0x7fca241593d0 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fca2406e4a0 con 0x7fca24152bf0 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca28e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fca2406e6f0 con 0x7fca240c6470 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca1be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca21e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca22800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca241593d0 0x7fca241f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca21e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41676/0 (socket says 172.21.15.92:41676) 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca21e00640 1 -- 172.21.15.92:0/1870791845 learned_addr learned my addr 172.21.15.92:0/1870791845 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:22.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.358+0000 7fca1be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46110/0 (socket says 172.21.15.92:46110) 2024-03-19T02:55:22.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 -- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 msgr2=0x7fca2415bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 -- 172.21.15.92:0/1870791845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca241593d0 msgr2=0x7fca241f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca241593d0 0x7fca241f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 -- 172.21.15.92:0/1870791845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca241f3930 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca21e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1be00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fca0c033000 tx=0x7fca0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c004220 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c031ec0 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c003610 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca241e4bb0 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.359+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca241f3ed0 con 0x7fca24152bf0 2024-03-19T02:55:22.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.360+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca240684c0 con 0x7fca24152bf0 2024-03-19T02:55:22.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.361+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fca0c0037b0 con 0x7fca24152bf0 2024-03-19T02:55:22.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.362+0000 7fca1a000640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 0x7fca0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.362+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fca0c0ac420 con 0x7fca24152bf0 2024-03-19T02:55:22.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.362+0000 7fca21e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 0x7fca0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.363+0000 7fca21e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 0x7fca0006e090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca140112c0 tx=0x7fca14009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.366+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fca0c041070 con 0x7fca24152bf0 2024-03-19T02:55:22.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.562+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca240c3b50 con 0x7fca24152bf0 2024-03-19T02:55:22.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.563+0000 7fca1a000640 1 -- 172.21.15.92:0/1870791845 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fca0c02f050 con 0x7fca24152bf0 2024-03-19T02:55:22.564 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 msgr2=0x7fca0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 0x7fca0006e090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca140112c0 tx=0x7fca14009190 comp rx=0 tx=0).stop 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 msgr2=0x7fca24148500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fca0c033000 tx=0x7fca0c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 shutdown_connections 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fca0006bbd0 0x7fca0006e090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca241593d0 0x7fca241f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca24152bf0 0x7fca24148500 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 --2- 172.21.15.92:0/1870791845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca240c6470 0x7fca2415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 >> 172.21.15.92:0/1870791845 conn(0x7fca240c8d90 msgr2=0x7fca240c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:22.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 shutdown_connections 2024-03-19T02:55:22.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.565+0000 7fca28e00640 1 -- 172.21.15.92:0/1870791845 wait complete. 2024-03-19T02:55:22.585 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-03-19T02:55:22.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:22.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.791+0000 7f036ee00640 1 Processor -- start 2024-03-19T02:55:22.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.791+0000 7f036ee00640 1 -- start start 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.793+0000 7f036ee00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f03680c6470 0x7f03680c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.793+0000 7f036ee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03680c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.793+0000 7f036ee00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03680c9a80 con 0x7f0368152bf0 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.793+0000 7f036ee00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f03680c6c90 con 0x7f03680c6470 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.793+0000 7f036ee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0368159910 con 0x7f03681593d0 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f03680c6470 0x7f03680c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f036ce00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03680c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f03680c6470 0x7f03680c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46118/0 (socket says 172.21.15.92:46118) 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0368152bf0 0x7f036806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46222/0 (socket says 172.21.15.92:46222) 2024-03-19T02:55:22.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 -- 172.21.15.92:0/465828390 learned_addr learned my addr 172.21.15.92:0/465828390 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:22.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 -- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 msgr2=0x7f03680c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 --2- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03680c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 -- 172.21.15.92:0/465828390 >> v1:172.21.15.92:6789/0 conn(0x7f0368152bf0 legacy=0x7f036806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.794+0000 7f0367400640 1 -- 172.21.15.92:0/465828390 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03680c8650 con 0x7f03680c6470 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.795+0000 7f0367400640 1 --2- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03680c3f80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0354009870 tx=0x7f035402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d0137d461ed3683 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.795+0000 7f0366a00640 1 -- 172.21.15.92:0/465828390 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354035020 con 0x7f03680c6470 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f0366a00640 1 -- 172.21.15.92:0/465828390 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0354031640 con 0x7f03680c6470 2024-03-19T02:55:22.797 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f0366a00640 1 -- 172.21.15.92:0/465828390 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035402d040 con 0x7f03680c6470 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 -- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 msgr2=0x7f03680c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 --2- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03680c3f80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0354009870 tx=0x7f035402b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 -- 172.21.15.92:0/465828390 shutdown_connections 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 --2- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03680c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 --2- 172.21.15.92:0/465828390 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03680c3f80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 -- 172.21.15.92:0/465828390 >> 172.21.15.92:0/465828390 conn(0x7f03680c8d90 msgr2=0x7f03680c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 -- 172.21.15.92:0/465828390 shutdown_connections 2024-03-19T02:55:22.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.796+0000 7f036ee00640 1 -- 172.21.15.92:0/465828390 wait complete. 2024-03-19T02:55:22.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.798+0000 7f036ee00640 1 Processor -- start 2024-03-19T02:55:22.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.798+0000 7f036ee00640 1 -- start start 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03681ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0368152bf0 0x7f03681f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f036806ea60 con 0x7f0368152bf0 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f036806e4a0 con 0x7f03680c6470 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f036806e6f0 con 0x7f03681593d0 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ce00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f0367400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0368152bf0 0x7f03681f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f0367e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03681ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ce00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41696/0 (socket says 172.21.15.92:41696) 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.799+0000 7f036ce00640 1 -- 172.21.15.92:0/290058657 learned_addr learned my addr 172.21.15.92:0/290058657 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 -- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 msgr2=0x7f03681ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03681ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 -- 172.21.15.92:0/290058657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0368152bf0 msgr2=0x7f03681f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0368152bf0 0x7f03681f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:22.802 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 -- 172.21.15.92:0/290058657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03681f2a70 con 0x7f03681593d0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ce00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f036000efc0 tx=0x7f036000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360018020 con 0x7f03681593d0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0360014480 con 0x7f03681593d0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360009e60 con 0x7f03681593d0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03681f2d50 con 0x7f03681593d0 2024-03-19T02:55:22.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.800+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03681f3310 con 0x7f03681593d0 2024-03-19T02:55:22.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.802+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03680684c0 con 0x7f03681593d0 2024-03-19T02:55:22.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.803+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f036001b020 con 0x7f03681593d0 2024-03-19T02:55:22.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.803+0000 7f035fe00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:22.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.804+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f036008b670 con 0x7f03681593d0 2024-03-19T02:55:22.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.804+0000 7f0367e00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:22.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.804+0000 7f0367e00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 0x7f034806e090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0358007fd0 tx=0x7f0358007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:22.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:22.808+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0360022020 con 0x7f03681593d0 2024-03-19T02:55:23.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.047+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03680c3b50 con 0x7f03681593d0 2024-03-19T02:55:23.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.048+0000 7f035fe00640 1 -- 172.21.15.92:0/290058657 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f0360010040 con 0x7f03681593d0 2024-03-19T02:55:23.049 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:23.050 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:55:23.050 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.050+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 msgr2=0x7f034806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.050+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 0x7f034806e090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0358007fd0 tx=0x7f0358007e80 comp rx=0 tx=0).stop 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.050+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 msgr2=0x7f03681f46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.050+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f036000efc0 tx=0x7f036000c490 comp rx=0 tx=0).stop 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 shutdown_connections 2024-03-19T02:55:23.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03681593d0 0x7f03681f46c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0368152bf0 0x7f03681f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 --2- 172.21.15.92:0/290058657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03680c6470 0x7f03681ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 >> 172.21.15.92:0/290058657 conn(0x7f03680c8d90 msgr2=0x7f0368148c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 shutdown_connections 2024-03-19T02:55:23.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.051+0000 7f036ee00640 1 -- 172.21.15.92:0/290058657 wait complete. 2024-03-19T02:55:23.071 INFO:tasks.mds_thrash:Ready to start thrashing 2024-03-19T02:55:23.071 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-03-19T02:55:23.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:23.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.267+0000 7f6302000640 1 Processor -- start 2024-03-19T02:55:23.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.267+0000 7f6302000640 1 -- start start 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.268+0000 7f6302000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62fc151900 0x7f62fc158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.268+0000 7f6302000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 0x7f62fc0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f6302000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc0c7da0 0x7f62fc0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f6302000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc158860 con 0x7f62fc151900 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f6302000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc0c3b80 con 0x7f62fc151ce0 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f6302000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc072e70 con 0x7f62fc0c7da0 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62fc151900 0x7f62fc158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62fc151900 0x7f62fc158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49550/0 (socket says 172.21.15.92:49550) 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 -- 172.21.15.92:0/3383160291 learned_addr learned my addr 172.21.15.92:0/3383160291 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62faa00640 1 --2- 172.21.15.92:0/3383160291 >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 0x7f62fc0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fbe00640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc0c7da0 0x7f62fc0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 -- 172.21.15.92:0/3383160291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc0c7da0 msgr2=0x7f62fc0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:23.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc0c7da0 0x7f62fc0c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 -- 172.21.15.92:0/3383160291 >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 msgr2=0x7f62fc0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:23.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 --2- 172.21.15.92:0/3383160291 >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 0x7f62fc0723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62fb400640 1 -- 172.21.15.92:0/3383160291 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc073150 con 0x7f62fc151900 2024-03-19T02:55:23.272 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.269+0000 7f62faa00640 1 --2- 172.21.15.92:0/3383160291 >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 0x7f62fc0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.270+0000 7f62fb400640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc151900 0x7f62fc158320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0c4a30 tx=0x7f62ec02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fbb8179ffb0ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.271+0000 7f62fa000640 1 -- 172.21.15.92:0/3383160291 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec034070 con 0x7f62fc151900 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.271+0000 7f62fa000640 1 -- 172.21.15.92:0/3383160291 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec030850 con 0x7f62fc151900 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.271+0000 7f62fa000640 1 -- 172.21.15.92:0/3383160291 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec030c20 con 0x7f62fc151900 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.271+0000 7f6302000640 1 -- 172.21.15.92:0/3383160291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc151900 msgr2=0x7f62fc158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.271+0000 7f6302000640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc151900 0x7f62fc158320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0c4a30 tx=0x7f62ec02b990 comp rx=0 tx=0).stop 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 -- 172.21.15.92:0/3383160291 shutdown_connections 2024-03-19T02:55:23.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc0c7da0 0x7f62fc0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 --2- 172.21.15.92:0/3383160291 >> v2:172.21.15.159:3300/0 conn(0x7f62fc151ce0 0x7f62fc0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 --2- 172.21.15.92:0/3383160291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc151900 0x7f62fc158320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 -- 172.21.15.92:0/3383160291 >> 172.21.15.92:0/3383160291 conn(0x7f62fc0c8d90 msgr2=0x7f62fc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 -- 172.21.15.92:0/3383160291 shutdown_connections 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.272+0000 7f6302000640 1 -- 172.21.15.92:0/3383160291 wait complete. 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.273+0000 7f6302000640 1 Processor -- start 2024-03-19T02:55:23.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.273+0000 7f6302000640 1 -- start start 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc0c7da0 0x7f62fc1ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc151ce0 0x7f62fc1decc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc072910 con 0x7f62fc0c7da0 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc071eb0 con 0x7f62fc151900 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.274+0000 7f6302000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc0c9af0 con 0x7f62fc151ce0 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62fbe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc151ce0 0x7f62fc1decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46138/0 (socket says 172.21.15.92:46138) 2024-03-19T02:55:23.277 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 -- 172.21.15.92:0/3069312731 learned_addr learned my addr 172.21.15.92:0/3069312731 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62fb400640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc0c7da0 0x7f62fc1ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 -- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc151ce0 msgr2=0x7f62fc1decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc151ce0 0x7f62fc1decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 -- 172.21.15.92:0/3069312731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc0c7da0 msgr2=0x7f62fc1ed670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc0c7da0 0x7f62fc1ed670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 -- 172.21.15.92:0/3069312731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc0cbf60 con 0x7f62fc151900 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62faa00640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62e800ebe0 tx=0x7f62e800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.275+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8017070 con 0x7f62fc151900 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.276+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e800cc30 con 0x7f62fc151900 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.276+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8013470 con 0x7f62fc151900 2024-03-19T02:55:23.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.276+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62fc1e0340 con 0x7f62fc151900 2024-03-19T02:55:23.279 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.276+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62fc1f0160 con 0x7f62fc151900 2024-03-19T02:55:23.279 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.277+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62fc0c60e0 con 0x7f62fc151900 2024-03-19T02:55:23.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.278+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f62e801a070 con 0x7f62fc151900 2024-03-19T02:55:23.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.278+0000 7f62f3400640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 0x7f62dc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.279+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f62e808a990 con 0x7f62fc151900 2024-03-19T02:55:23.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.279+0000 7f62fb400640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 0x7f62dc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.280 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.279+0000 7f62fb400640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 0x7f62dc06e0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f62ec036000 tx=0x7f62ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.284 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.283+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f62fc0c60e0 con 0x7f62fc151900 2024-03-19T02:55:23.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.474+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62fc0684c0 con 0x7f62dc06bc00 2024-03-19T02:55:23.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.482+0000 7f62f3400640 1 -- 172.21.15.92:0/3069312731 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7f62fc0684c0 con 0x7f62dc06bc00 2024-03-19T02:55:23.487 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:23.488 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:23.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 msgr2=0x7f62dc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 0x7f62dc06e0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f62ec036000 tx=0x7f62ec008030 comp rx=0 tx=0).stop 2024-03-19T02:55:23.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 msgr2=0x7f62fc0cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62e800ebe0 tx=0x7f62e800c520 comp rx=0 tx=0).stop 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 shutdown_connections 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f62dc06bc00 0x7f62dc06e0c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62fc151ce0 0x7f62fc1decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62fc151900 0x7f62fc0cb350 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 --2- 172.21.15.92:0/3069312731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62fc0c7da0 0x7f62fc1ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 >> 172.21.15.92:0/3069312731 conn(0x7f62fc0c8d90 msgr2=0x7f62fc0c8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 shutdown_connections 2024-03-19T02:55:23.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.488+0000 7f6302000640 1 -- 172.21.15.92:0/3069312731 wait complete. 2024-03-19T02:55:23.507 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-19T02:55:21.841416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593032,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217920,"kb_used_data":1792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756800,"statfs":{"total":773094113280,"available":772870963200,"internally_reserved":0,"allocated":1835008,"data_stored":1753536,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002078"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380801+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380801+0000","last_peered":"2024-03-19T02:54:57.380801+0000","last_clean":"2024-03-19T02:54:57.380801+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380801+0000","last_undegraded":"2024-03-19T02:54:57.380801+0000","last_fullsized":"2024-03-19T02:54:57.380801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081567+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081567+0000","last_peered":"2024-03-19T02:54:59.081567+0000","last_clean":"2024-03-19T02:54:59.081567+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081567+0000","last_undegraded":"2024-03-19T02:54:59.081567+0000","last_fullsized":"2024-03-19T02:54:59.081567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.182131+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:55:01.182131+0000","last_peered":"2024-03-19T02:55:01.182131+0000","last_clean":"2024-03-19T02:55:01.182131+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:55:01.182131+0000","last_undegraded":"2024-03-19T02:55:01.182131+0000","last_fullsized":"2024-03-19T02:55:01.182131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:17.259448+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:55:17.259448+0000","last_peered":"2024-03-19T02:55:17.259448+0000","last_clean":"2024-03-19T02:55:17.259448+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:55:17.259448+0000","last_undegraded":"2024-03-19T02:55:17.259448+0000","last_fullsized":"2024-03-19T02:55:17.259448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183309+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:55:01.183309+0000","last_peered":"2024-03-19T02:55:01.183309+0000","last_clean":"2024-03-19T02:55:01.183309+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:55:01.183309+0000","last_undegraded":"2024-03-19T02:55:01.183309+0000","last_fullsized":"2024-03-19T02:55:01.183309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.186421+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:55:01.186421+0000","last_peered":"2024-03-19T02:55:01.186421+0000","last_clean":"2024-03-19T02:55:01.186421+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:55:01.186421+0000","last_undegraded":"2024-03-19T02:55:01.186421+0000","last_fullsized":"2024-03-19T02:55:01.186421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197959+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197959+0000","last_peered":"2024-03-19T02:54:57.197959+0000","last_clean":"2024-03-19T02:54:57.197959+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197959+0000","last_undegraded":"2024-03-19T02:54:57.197959+0000","last_fullsized":"2024-03-19T02:54:57.197959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188015+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:55:01.188015+0000","last_peered":"2024-03-19T02:55:01.188015+0000","last_clean":"2024-03-19T02:55:01.188015+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:55:01.188015+0000","last_undegraded":"2024-03-19T02:55:01.188015+0000","last_fullsized":"2024-03-19T02:55:01.188015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201693+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201693+0000","last_peered":"2024-03-19T02:54:57.201693+0000","last_clean":"2024-03-19T02:54:57.201693+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201693+0000","last_undegraded":"2024-03-19T02:54:57.201693+0000","last_fullsized":"2024-03-19T02:54:57.201693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201786+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201786+0000","last_peered":"2024-03-19T02:54:57.201786+0000","last_clean":"2024-03-19T02:54:57.201786+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201786+0000","last_undegraded":"2024-03-19T02:54:57.201786+0000","last_fullsized":"2024-03-19T02:54:57.201786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177520+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:55:01.177520+0000","last_peered":"2024-03-19T02:55:01.177520+0000","last_clean":"2024-03-19T02:55:01.177520+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:55:01.177520+0000","last_undegraded":"2024-03-19T02:55:01.177520+0000","last_fullsized":"2024-03-19T02:55:01.177520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177763+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:55:01.177763+0000","last_peered":"2024-03-19T02:55:01.177763+0000","last_clean":"2024-03-19T02:55:01.177763+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:55:01.177763+0000","last_undegraded":"2024-03-19T02:55:01.177763+0000","last_fullsized":"2024-03-19T02:55:01.177763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177777+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:55:01.177777+0000","last_peered":"2024-03-19T02:55:01.177777+0000","last_clean":"2024-03-19T02:55:01.177777+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:55:01.177777+0000","last_undegraded":"2024-03-19T02:55:01.177777+0000","last_fullsized":"2024-03-19T02:55:01.177777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.178273+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:55:01.178273+0000","last_peered":"2024-03-19T02:55:01.178273+0000","last_clean":"2024-03-19T02:55:01.178273+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:55:01.178273+0000","last_undegraded":"2024-03-19T02:55:01.178273+0000","last_fullsized":"2024-03-19T02:55:01.178273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381552+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381552+0000","last_peered":"2024-03-19T02:54:57.381552+0000","last_clean":"2024-03-19T02:54:57.381552+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381552+0000","last_undegraded":"2024-03-19T02:54:57.381552+0000","last_fullsized":"2024-03-19T02:54:57.381552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188465+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:55:01.188465+0000","last_peered":"2024-03-19T02:55:01.188465+0000","last_clean":"2024-03-19T02:55:01.188465+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:55:01.188465+0000","last_undegraded":"2024-03-19T02:55:01.188465+0000","last_fullsized":"2024-03-19T02:55:01.188465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2664,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661884,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":661940,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:23.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:23.762 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.761+0000 7ff14e200640 1 Processor -- start 2024-03-19T02:55:23.763 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.761+0000 7ff14e200640 1 -- start start 2024-03-19T02:55:23.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.762+0000 7ff14e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 0x7ff148072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.762+0000 7ff14e200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff1480c5860 0x7ff1481599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.762+0000 7ff14e200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff148072140 con 0x7ff1480c36f0 2024-03-19T02:55:23.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.762+0000 7ff14e200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff148159520 con 0x7ff1480c5860 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.762+0000 7ff14e200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff1480c4ef0 con 0x7ff1480c5480 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff1480c5860 0x7ff1481599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff14ce00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 0x7ff148072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff1480c5860 0x7ff1481599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46152/0 (socket says 172.21.15.92:46152) 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 -- 172.21.15.92:0/3102389393 learned_addr learned my addr 172.21.15.92:0/3102389393 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:23.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 -- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 msgr2=0x7ff148072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:23.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 0x7ff148072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 -- 172.21.15.92:0/3102389393 >> v1:172.21.15.92:6789/0 conn(0x7ff1480c36f0 legacy=0x7ff148151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147e00640 1 -- 172.21.15.92:0/3102389393 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1480728d0 con 0x7ff1480c5860 2024-03-19T02:55:23.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff14ce00640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 0x7ff148072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:23.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.763+0000 7ff147400640 1 -- 172.21.15.92:0/3102389393 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223918403 0 0) 0x7ff148072140 con 0x7ff1480c36f0 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.764+0000 7ff147e00640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5860 0x7ff1481599b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff130009870 tx=0x7ff13002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0499c44e041e116 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.764+0000 7ff147400640 1 -- 172.21.15.92:0/3102389393 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130035020 con 0x7ff1480c5860 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff147400640 1 -- 172.21.15.92:0/3102389393 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff130031640 con 0x7ff1480c5860 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff147400640 1 -- 172.21.15.92:0/3102389393 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13002d040 con 0x7ff1480c5860 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 -- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5860 msgr2=0x7ff1481599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5860 0x7ff1481599b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff130009870 tx=0x7ff13002b8f0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 -- 172.21.15.92:0/3102389393 shutdown_connections 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5860 0x7ff1481599b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 --2- 172.21.15.92:0/3102389393 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c5480 0x7ff148072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 -- 172.21.15.92:0/3102389393 >> 172.21.15.92:0/3102389393 conn(0x7ff1480cc1f0 msgr2=0x7ff1480cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 -- 172.21.15.92:0/3102389393 shutdown_connections 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.765+0000 7ff14e200640 1 -- 172.21.15.92:0/3102389393 wait complete. 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.767+0000 7ff14e200640 1 Processor -- start 2024-03-19T02:55:23.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.767+0000 7ff14e200640 1 -- start start 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c36f0 0x7ff148158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1480c5860 0x7ff14814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff148158ed0 con 0x7ff1480c5860 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff148071e60 con 0x7ff1480c5480 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14e200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff1481591b0 con 0x7ff1480c36f0 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff147e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff147e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46166/0 (socket says 172.21.15.92:46166) 2024-03-19T02:55:23.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff147e00640 1 -- 172.21.15.92:0/1203793484 learned_addr learned my addr 172.21.15.92:0/1203793484 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14ce00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c36f0 0x7ff148158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.768+0000 7ff14d800640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1480c5860 0x7ff14814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 -- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c36f0 msgr2=0x7ff148158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c36f0 0x7ff148158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 -- 172.21.15.92:0/1203793484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1480c5860 msgr2=0x7ff14814cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1480c5860 0x7ff14814cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 -- 172.21.15.92:0/1203793484 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14814fc00 con 0x7ff1480c5480 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff147e00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff13002b930 tx=0x7ff1300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130040070 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13003e070 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130035040 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1481f3a00 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.769+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1481f3fc0 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.770+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14804f580 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.771+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff130038020 con 0x7ff1480c5480 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.772+0000 7ff144c00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 0x7ff12406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:23.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.772+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff1300ac250 con 0x7ff1480c5480 2024-03-19T02:55:23.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.772+0000 7ff14ce00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 0x7ff12406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:23.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.773+0000 7ff14ce00640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 0x7ff12406e090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff138009560 tx=0x7ff138009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:23.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.776+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff13007a6e0 con 0x7ff1480c5480 2024-03-19T02:55:23.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.971+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff148156100 con 0x7ff12406bbd0 2024-03-19T02:55:23.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.979+0000 7ff144c00640 1 -- 172.21.15.92:0/1203793484 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7ff148156100 con 0x7ff12406bbd0 2024-03-19T02:55:23.986 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:23.987 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.985+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 msgr2=0x7ff12406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.985+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 0x7ff12406e090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff138009560 tx=0x7ff138009190 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 msgr2=0x7ff1480c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff13002b930 tx=0x7ff1300046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 shutdown_connections 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff12406bbd0 0x7ff12406e090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1480c5860 0x7ff14814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1480c5480 0x7ff1480c6a00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 --2- 172.21.15.92:0/1203793484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1480c36f0 0x7ff148158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 >> 172.21.15.92:0/1203793484 conn(0x7ff1480cc1f0 msgr2=0x7ff14815a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:23.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 shutdown_connections 2024-03-19T02:55:23.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:23.986+0000 7ff14e200640 1 -- 172.21.15.92:0/1203793484 wait complete. 2024-03-19T02:55:24.005 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-19T02:55:23.841859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593032,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217920,"kb_used_data":1792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756800,"statfs":{"total":773094113280,"available":772870963200,"internally_reserved":0,"allocated":1835008,"data_stored":1753536,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002222"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380801+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380801+0000","last_peered":"2024-03-19T02:54:57.380801+0000","last_clean":"2024-03-19T02:54:57.380801+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380801+0000","last_undegraded":"2024-03-19T02:54:57.380801+0000","last_fullsized":"2024-03-19T02:54:57.380801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081567+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081567+0000","last_peered":"2024-03-19T02:54:59.081567+0000","last_clean":"2024-03-19T02:54:59.081567+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081567+0000","last_undegraded":"2024-03-19T02:54:59.081567+0000","last_fullsized":"2024-03-19T02:54:59.081567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.182131+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:55:01.182131+0000","last_peered":"2024-03-19T02:55:01.182131+0000","last_clean":"2024-03-19T02:55:01.182131+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:55:01.182131+0000","last_undegraded":"2024-03-19T02:55:01.182131+0000","last_fullsized":"2024-03-19T02:55:01.182131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:17.259448+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:55:17.259448+0000","last_peered":"2024-03-19T02:55:17.259448+0000","last_clean":"2024-03-19T02:55:17.259448+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:55:17.259448+0000","last_undegraded":"2024-03-19T02:55:17.259448+0000","last_fullsized":"2024-03-19T02:55:17.259448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183309+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:55:01.183309+0000","last_peered":"2024-03-19T02:55:01.183309+0000","last_clean":"2024-03-19T02:55:01.183309+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:55:01.183309+0000","last_undegraded":"2024-03-19T02:55:01.183309+0000","last_fullsized":"2024-03-19T02:55:01.183309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.186421+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:55:01.186421+0000","last_peered":"2024-03-19T02:55:01.186421+0000","last_clean":"2024-03-19T02:55:01.186421+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:55:01.186421+0000","last_undegraded":"2024-03-19T02:55:01.186421+0000","last_fullsized":"2024-03-19T02:55:01.186421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197959+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197959+0000","last_peered":"2024-03-19T02:54:57.197959+0000","last_clean":"2024-03-19T02:54:57.197959+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197959+0000","last_undegraded":"2024-03-19T02:54:57.197959+0000","last_fullsized":"2024-03-19T02:54:57.197959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188015+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:55:01.188015+0000","last_peered":"2024-03-19T02:55:01.188015+0000","last_clean":"2024-03-19T02:55:01.188015+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:55:01.188015+0000","last_undegraded":"2024-03-19T02:55:01.188015+0000","last_fullsized":"2024-03-19T02:55:01.188015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201693+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201693+0000","last_peered":"2024-03-19T02:54:57.201693+0000","last_clean":"2024-03-19T02:54:57.201693+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201693+0000","last_undegraded":"2024-03-19T02:54:57.201693+0000","last_fullsized":"2024-03-19T02:54:57.201693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201786+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201786+0000","last_peered":"2024-03-19T02:54:57.201786+0000","last_clean":"2024-03-19T02:54:57.201786+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201786+0000","last_undegraded":"2024-03-19T02:54:57.201786+0000","last_fullsized":"2024-03-19T02:54:57.201786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177520+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:55:01.177520+0000","last_peered":"2024-03-19T02:55:01.177520+0000","last_clean":"2024-03-19T02:55:01.177520+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:55:01.177520+0000","last_undegraded":"2024-03-19T02:55:01.177520+0000","last_fullsized":"2024-03-19T02:55:01.177520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177763+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:55:01.177763+0000","last_peered":"2024-03-19T02:55:01.177763+0000","last_clean":"2024-03-19T02:55:01.177763+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:55:01.177763+0000","last_undegraded":"2024-03-19T02:55:01.177763+0000","last_fullsized":"2024-03-19T02:55:01.177763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177777+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:55:01.177777+0000","last_peered":"2024-03-19T02:55:01.177777+0000","last_clean":"2024-03-19T02:55:01.177777+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:55:01.177777+0000","last_undegraded":"2024-03-19T02:55:01.177777+0000","last_fullsized":"2024-03-19T02:55:01.177777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.178273+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:55:01.178273+0000","last_peered":"2024-03-19T02:55:01.178273+0000","last_clean":"2024-03-19T02:55:01.178273+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:55:01.178273+0000","last_undegraded":"2024-03-19T02:55:01.178273+0000","last_fullsized":"2024-03-19T02:55:01.178273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381552+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381552+0000","last_peered":"2024-03-19T02:54:57.381552+0000","last_clean":"2024-03-19T02:54:57.381552+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381552+0000","last_undegraded":"2024-03-19T02:54:57.381552+0000","last_fullsized":"2024-03-19T02:54:57.381552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188465+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:55:01.188465+0000","last_peered":"2024-03-19T02:55:01.188465+0000","last_clean":"2024-03-19T02:55:01.188465+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:55:01.188465+0000","last_undegraded":"2024-03-19T02:55:01.188465+0000","last_fullsized":"2024-03-19T02:55:01.188465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2664,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661884,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":661940,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:24.009 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-03-19T02:55:24.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:55:24.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.256+0000 7f3768a00640 1 Processor -- start 2024-03-19T02:55:24.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.256+0000 7f3768a00640 1 -- start start 2024-03-19T02:55:24.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.257+0000 7f3768a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3764152bf0 0x7f376406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.257+0000 7f3768a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37641593d0 0x7f37640c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.259 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.257+0000 7f3768a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37640c9a80 con 0x7f37640c6470 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.257+0000 7f3768a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f37640c6c90 con 0x7f3764152bf0 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.257+0000 7f3768a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3764159910 con 0x7f37641593d0 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3764152bf0 0x7f376406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3759400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f37640c6470 0x7f37640c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49380/0 (socket says 172.21.15.92:49380) 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3759400640 1 -- 172.21.15.92:0/1391904670 learned_addr learned my addr 172.21.15.92:0/1391904670 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3762800640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37641593d0 0x7f37640c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761400640 1 -- 172.21.15.92:0/1391904670 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235191376 0 0) 0x7f37640c9a80 con 0x7f37640c6470 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761400640 1 -- 172.21.15.92:0/1391904670 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3744003520 con 0x7f37640c6470 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761e00640 1 -- 172.21.15.92:0/1391904670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37641593d0 msgr2=0x7f37640c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761e00640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37641593d0 0x7f37640c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761e00640 1 -- 172.21.15.92:0/1391904670 >> v1:172.21.15.159:6789/0 conn(0x7f37640c6470 legacy=0x7f37640c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.260 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.258+0000 7f3761e00640 1 -- 172.21.15.92:0/1391904670 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37640c8650 con 0x7f3764152bf0 2024-03-19T02:55:24.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.259+0000 7f3761e00640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3764152bf0 0x7f376406eef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37540097a0 tx=0x7f375402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbfe5144bb31c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3761400640 1 -- 172.21.15.92:0/1391904670 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754031780 con 0x7f3764152bf0 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3761400640 1 -- 172.21.15.92:0/1391904670 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3754037070 con 0x7f3764152bf0 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3761400640 1 -- 172.21.15.92:0/1391904670 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754031b20 con 0x7f3764152bf0 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 -- 172.21.15.92:0/1391904670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3764152bf0 msgr2=0x7f376406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3764152bf0 0x7f376406eef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37540097a0 tx=0x7f375402b820 comp rx=0 tx=0).stop 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 -- 172.21.15.92:0/1391904670 shutdown_connections 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37641593d0 0x7f37640c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 --2- 172.21.15.92:0/1391904670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3764152bf0 0x7f376406eef0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 -- 172.21.15.92:0/1391904670 >> 172.21.15.92:0/1391904670 conn(0x7f37640c8d90 msgr2=0x7f37640c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 -- 172.21.15.92:0/1391904670 shutdown_connections 2024-03-19T02:55:24.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.260+0000 7f3768a00640 1 -- 172.21.15.92:0/1391904670 wait complete. 2024-03-19T02:55:24.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.262+0000 7f3768a00640 1 Processor -- start 2024-03-19T02:55:24.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.262+0000 7f3768a00640 1 -- start start 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37641593d0 0x7f37641f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f376406ea60 con 0x7f37641593d0 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f376406e4a0 con 0x7f37640c6470 2024-03-19T02:55:24.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3768a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f376406e6f0 con 0x7f3764152bf0 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3761e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3759400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3761e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46168/0 (socket says 172.21.15.92:46168) 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.263+0000 7f3759400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41766/0 (socket says 172.21.15.92:41766) 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 -- 172.21.15.92:0/3675337911 learned_addr learned my addr 172.21.15.92:0/3675337911 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3762800640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37641593d0 0x7f37641f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 -- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 msgr2=0x7f37641e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 -- 172.21.15.92:0/3675337911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37641593d0 msgr2=0x7f37641f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37641593d0 0x7f37641f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 -- 172.21.15.92:0/3675337911 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3764071ec0 con 0x7f37640c6470 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3759400640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f3761e00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f375402be30 tx=0x7f37540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.264+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754032410 con 0x7f37640c6470 2024-03-19T02:55:24.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.265+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37540325b0 con 0x7f37640c6470 2024-03-19T02:55:24.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.265+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37641584f0 con 0x7f37640c6470 2024-03-19T02:55:24.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.265+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754032960 con 0x7f37640c6470 2024-03-19T02:55:24.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.265+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376414ca40 con 0x7f37640c6470 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.266+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37640684c0 con 0x7f37640c6470 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.267+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3754032b00 con 0x7f37640c6470 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.268+0000 7f375aa00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 0x7f374006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.268+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f37540ac4e0 con 0x7f37640c6470 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.268+0000 7f3759400640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 0x7f374006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.268+0000 7f3759400640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 0x7f374006e090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37640c78a0 tx=0x7f374c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.273 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.272+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3754040070 con 0x7f37640c6470 2024-03-19T02:55:24.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.464+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37641582a0 con 0x7f374006bbd0 2024-03-19T02:55:24.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.477+0000 7f375aa00640 1 -- 172.21.15.92:0/3675337911 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7f37641582a0 con 0x7f374006bbd0 2024-03-19T02:55:24.481 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:24.484 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 msgr2=0x7f374006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 0x7f374006e090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37640c78a0 tx=0x7f374c009040 comp rx=0 tx=0).stop 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 msgr2=0x7f37641e48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f375402be30 tx=0x7f37540046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 shutdown_connections 2024-03-19T02:55:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f374006bbd0 0x7f374006e090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37641593d0 0x7f37641f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3764152bf0 0x7f37641e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 --2- 172.21.15.92:0/3675337911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37640c6470 0x7f37641e48b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 >> 172.21.15.92:0/3675337911 conn(0x7f37640c8d90 msgr2=0x7f37640c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 shutdown_connections 2024-03-19T02:55:24.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.484+0000 7f3768a00640 1 -- 172.21.15.92:0/3675337911 wait complete. 2024-03-19T02:55:24.505 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-19T02:55:23.841859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593032,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217920,"kb_used_data":1792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756800,"statfs":{"total":773094113280,"available":772870963200,"internally_reserved":0,"allocated":1835008,"data_stored":1753536,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002222"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081894+0000","last_change":"2024-03-19T02:54:56.198012+0000","last_active":"2024-03-19T02:54:59.081894+0000","last_peered":"2024-03-19T02:54:59.081894+0000","last_clean":"2024-03-19T02:54:59.081894+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T02:54:59.081894+0000","last_undegraded":"2024-03-19T02:54:59.081894+0000","last_fullsized":"2024-03-19T02:54:59.081894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:26.496423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201940+0000","last_change":"2024-03-19T02:54:56.213792+0000","last_active":"2024-03-19T02:54:57.201940+0000","last_peered":"2024-03-19T02:54:57.201940+0000","last_clean":"2024-03-19T02:54:57.201940+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T02:54:57.201940+0000","last_undegraded":"2024-03-19T02:54:57.201940+0000","last_fullsized":"2024-03-19T02:54:57.201940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:48.236251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083880+0000","last_change":"2024-03-19T02:54:56.212815+0000","last_active":"2024-03-19T02:54:59.083880+0000","last_peered":"2024-03-19T02:54:59.083880+0000","last_clean":"2024-03-19T02:54:59.083880+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T02:54:59.083880+0000","last_undegraded":"2024-03-19T02:54:59.083880+0000","last_fullsized":"2024-03-19T02:54:59.083880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.486503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082981+0000","last_change":"2024-03-19T02:54:56.195638+0000","last_active":"2024-03-19T02:54:59.082981+0000","last_peered":"2024-03-19T02:54:59.082981+0000","last_clean":"2024-03-19T02:54:59.082981+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T02:54:59.082981+0000","last_undegraded":"2024-03-19T02:54:59.082981+0000","last_fullsized":"2024-03-19T02:54:59.082981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.989996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082145+0000","last_change":"2024-03-19T02:54:56.194481+0000","last_active":"2024-03-19T02:54:59.082145+0000","last_peered":"2024-03-19T02:54:59.082145+0000","last_clean":"2024-03-19T02:54:59.082145+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T02:54:59.082145+0000","last_undegraded":"2024-03-19T02:54:59.082145+0000","last_fullsized":"2024-03-19T02:54:59.082145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:35.498808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082705+0000","last_change":"2024-03-19T02:54:56.213698+0000","last_active":"2024-03-19T02:54:59.082705+0000","last_peered":"2024-03-19T02:54:59.082705+0000","last_clean":"2024-03-19T02:54:59.082705+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T02:54:59.082705+0000","last_undegraded":"2024-03-19T02:54:59.082705+0000","last_fullsized":"2024-03-19T02:54:59.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:13.140895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081597+0000","last_change":"2024-03-19T02:54:56.199402+0000","last_active":"2024-03-19T02:54:59.081597+0000","last_peered":"2024-03-19T02:54:59.081597+0000","last_clean":"2024-03-19T02:54:59.081597+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T02:54:59.081597+0000","last_undegraded":"2024-03-19T02:54:59.081597+0000","last_fullsized":"2024-03-19T02:54:59.081597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:16.922535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380344+0000","last_change":"2024-03-19T02:54:56.205199+0000","last_active":"2024-03-19T02:54:57.380344+0000","last_peered":"2024-03-19T02:54:57.380344+0000","last_clean":"2024-03-19T02:54:57.380344+0000","last_became_active":"2024-03-19T02:54:56.203787+0000","last_became_peered":"2024-03-19T02:54:56.203787+0000","last_unstale":"2024-03-19T02:54:57.380344+0000","last_undegraded":"2024-03-19T02:54:57.380344+0000","last_fullsized":"2024-03-19T02:54:57.380344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:17.304632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082059+0000","last_change":"2024-03-19T02:54:56.198696+0000","last_active":"2024-03-19T02:54:59.082059+0000","last_peered":"2024-03-19T02:54:59.082059+0000","last_clean":"2024-03-19T02:54:59.082059+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T02:54:59.082059+0000","last_undegraded":"2024-03-19T02:54:59.082059+0000","last_fullsized":"2024-03-19T02:54:59.082059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:50.924347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081694+0000","last_change":"2024-03-19T02:54:56.197688+0000","last_active":"2024-03-19T02:54:59.081694+0000","last_peered":"2024-03-19T02:54:59.081694+0000","last_clean":"2024-03-19T02:54:59.081694+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T02:54:59.081694+0000","last_undegraded":"2024-03-19T02:54:59.081694+0000","last_fullsized":"2024-03-19T02:54:59.081694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:03.016073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081990+0000","last_change":"2024-03-19T02:54:56.197179+0000","last_active":"2024-03-19T02:54:59.081990+0000","last_peered":"2024-03-19T02:54:59.081990+0000","last_clean":"2024-03-19T02:54:59.081990+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T02:54:59.081990+0000","last_undegraded":"2024-03-19T02:54:59.081990+0000","last_fullsized":"2024-03-19T02:54:59.081990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:09.413711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198020+0000","last_change":"2024-03-19T02:54:56.187239+0000","last_active":"2024-03-19T02:54:57.198020+0000","last_peered":"2024-03-19T02:54:57.198020+0000","last_clean":"2024-03-19T02:54:57.198020+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T02:54:57.198020+0000","last_undegraded":"2024-03-19T02:54:57.198020+0000","last_fullsized":"2024-03-19T02:54:57.198020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:42.471179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380535+0000","last_change":"2024-03-19T02:54:56.191923+0000","last_active":"2024-03-19T02:54:57.380535+0000","last_peered":"2024-03-19T02:54:57.380535+0000","last_clean":"2024-03-19T02:54:57.380535+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T02:54:57.380535+0000","last_undegraded":"2024-03-19T02:54:57.380535+0000","last_fullsized":"2024-03-19T02:54:57.380535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:26.453736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083122+0000","last_change":"2024-03-19T02:54:56.209837+0000","last_active":"2024-03-19T02:54:59.083122+0000","last_peered":"2024-03-19T02:54:59.083122+0000","last_clean":"2024-03-19T02:54:59.083122+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083122+0000","last_undegraded":"2024-03-19T02:54:59.083122+0000","last_fullsized":"2024-03-19T02:54:59.083122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.449317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200957+0000","last_change":"2024-03-19T02:54:56.201095+0000","last_active":"2024-03-19T02:54:57.200957+0000","last_peered":"2024-03-19T02:54:57.200957+0000","last_clean":"2024-03-19T02:54:57.200957+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T02:54:57.200957+0000","last_undegraded":"2024-03-19T02:54:57.200957+0000","last_fullsized":"2024-03-19T02:54:57.200957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:22.938007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082440+0000","last_change":"2024-03-19T02:54:56.200960+0000","last_active":"2024-03-19T02:54:59.082440+0000","last_peered":"2024-03-19T02:54:59.082440+0000","last_clean":"2024-03-19T02:54:59.082440+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T02:54:59.082440+0000","last_undegraded":"2024-03-19T02:54:59.082440+0000","last_fullsized":"2024-03-19T02:54:59.082440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:30.071240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083416+0000","last_change":"2024-03-19T02:54:56.199879+0000","last_active":"2024-03-19T02:54:59.083416+0000","last_peered":"2024-03-19T02:54:59.083416+0000","last_clean":"2024-03-19T02:54:59.083416+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T02:54:59.083416+0000","last_undegraded":"2024-03-19T02:54:59.083416+0000","last_fullsized":"2024-03-19T02:54:59.083416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:37.631402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197594+0000","last_change":"2024-03-19T02:54:56.190137+0000","last_active":"2024-03-19T02:54:57.197594+0000","last_peered":"2024-03-19T02:54:57.197594+0000","last_clean":"2024-03-19T02:54:57.197594+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T02:54:57.197594+0000","last_undegraded":"2024-03-19T02:54:57.197594+0000","last_fullsized":"2024-03-19T02:54:57.197594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:50.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082405+0000","last_change":"2024-03-19T02:54:56.195208+0000","last_active":"2024-03-19T02:54:59.082405+0000","last_peered":"2024-03-19T02:54:59.082405+0000","last_clean":"2024-03-19T02:54:59.082405+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T02:54:59.082405+0000","last_undegraded":"2024-03-19T02:54:59.082405+0000","last_fullsized":"2024-03-19T02:54:59.082405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:25.012553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081484+0000","last_change":"2024-03-19T02:54:56.192905+0000","last_active":"2024-03-19T02:54:59.081484+0000","last_peered":"2024-03-19T02:54:59.081484+0000","last_clean":"2024-03-19T02:54:59.081484+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T02:54:59.081484+0000","last_undegraded":"2024-03-19T02:54:59.081484+0000","last_fullsized":"2024-03-19T02:54:59.081484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:19.237323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083615+0000","last_change":"2024-03-19T02:54:56.215174+0000","last_active":"2024-03-19T02:54:59.083615+0000","last_peered":"2024-03-19T02:54:59.083615+0000","last_clean":"2024-03-19T02:54:59.083615+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T02:54:59.083615+0000","last_undegraded":"2024-03-19T02:54:59.083615+0000","last_fullsized":"2024-03-19T02:54:59.083615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:10.509763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082777+0000","last_change":"2024-03-19T02:54:56.190322+0000","last_active":"2024-03-19T02:54:59.082777+0000","last_peered":"2024-03-19T02:54:59.082777+0000","last_clean":"2024-03-19T02:54:59.082777+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T02:54:59.082777+0000","last_undegraded":"2024-03-19T02:54:59.082777+0000","last_fullsized":"2024-03-19T02:54:59.082777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:27.510922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380286+0000","last_change":"2024-03-19T02:54:56.190591+0000","last_active":"2024-03-19T02:54:57.380286+0000","last_peered":"2024-03-19T02:54:57.380286+0000","last_clean":"2024-03-19T02:54:57.380286+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T02:54:57.380286+0000","last_undegraded":"2024-03-19T02:54:57.380286+0000","last_fullsized":"2024-03-19T02:54:57.380286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:30.543941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082587+0000","last_change":"2024-03-19T02:54:56.193284+0000","last_active":"2024-03-19T02:54:59.082587+0000","last_peered":"2024-03-19T02:54:59.082587+0000","last_clean":"2024-03-19T02:54:59.082587+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T02:54:59.082587+0000","last_undegraded":"2024-03-19T02:54:59.082587+0000","last_fullsized":"2024-03-19T02:54:59.082587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:48.219909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381210+0000","last_change":"2024-03-19T02:54:56.195409+0000","last_active":"2024-03-19T02:54:57.381210+0000","last_peered":"2024-03-19T02:54:57.381210+0000","last_clean":"2024-03-19T02:54:57.381210+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T02:54:57.381210+0000","last_undegraded":"2024-03-19T02:54:57.381210+0000","last_fullsized":"2024-03-19T02:54:57.381210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:45.570612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081858+0000","last_change":"2024-03-19T02:54:56.196111+0000","last_active":"2024-03-19T02:54:59.081858+0000","last_peered":"2024-03-19T02:54:59.081858+0000","last_clean":"2024-03-19T02:54:59.081858+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T02:54:59.081858+0000","last_undegraded":"2024-03-19T02:54:59.081858+0000","last_fullsized":"2024-03-19T02:54:59.081858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:13:57.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201514+0000","last_change":"2024-03-19T02:54:56.212772+0000","last_active":"2024-03-19T02:54:57.201514+0000","last_peered":"2024-03-19T02:54:57.201514+0000","last_clean":"2024-03-19T02:54:57.201514+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T02:54:57.201514+0000","last_undegraded":"2024-03-19T02:54:57.201514+0000","last_fullsized":"2024-03-19T02:54:57.201514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:30.402124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381273+0000","last_change":"2024-03-19T02:54:56.196493+0000","last_active":"2024-03-19T02:54:57.381273+0000","last_peered":"2024-03-19T02:54:57.381273+0000","last_clean":"2024-03-19T02:54:57.381273+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T02:54:57.381273+0000","last_undegraded":"2024-03-19T02:54:57.381273+0000","last_fullsized":"2024-03-19T02:54:57.381273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.166715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084030+0000","last_change":"2024-03-19T02:54:56.215472+0000","last_active":"2024-03-19T02:54:59.084030+0000","last_peered":"2024-03-19T02:54:59.084030+0000","last_clean":"2024-03-19T02:54:59.084030+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T02:54:59.084030+0000","last_undegraded":"2024-03-19T02:54:59.084030+0000","last_fullsized":"2024-03-19T02:54:59.084030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:06.735895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082901+0000","last_change":"2024-03-19T02:54:56.210786+0000","last_active":"2024-03-19T02:54:59.082901+0000","last_peered":"2024-03-19T02:54:59.082901+0000","last_clean":"2024-03-19T02:54:59.082901+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T02:54:59.082901+0000","last_undegraded":"2024-03-19T02:54:59.082901+0000","last_fullsized":"2024-03-19T02:54:59.082901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:20.552959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197113+0000","last_change":"2024-03-19T02:54:56.185344+0000","last_active":"2024-03-19T02:54:57.197113+0000","last_peered":"2024-03-19T02:54:57.197113+0000","last_clean":"2024-03-19T02:54:57.197113+0000","last_became_active":"2024-03-19T02:54:56.184993+0000","last_became_peered":"2024-03-19T02:54:56.184993+0000","last_unstale":"2024-03-19T02:54:57.197113+0000","last_undegraded":"2024-03-19T02:54:57.197113+0000","last_fullsized":"2024-03-19T02:54:57.197113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:02:30.691852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081175+0000","last_change":"2024-03-19T02:54:56.209977+0000","last_active":"2024-03-19T02:54:59.081175+0000","last_peered":"2024-03-19T02:54:59.081175+0000","last_clean":"2024-03-19T02:54:59.081175+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T02:54:59.081175+0000","last_undegraded":"2024-03-19T02:54:59.081175+0000","last_fullsized":"2024-03-19T02:54:59.081175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:22.229597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197656+0000","last_change":"2024-03-19T02:54:56.197064+0000","last_active":"2024-03-19T02:54:57.197656+0000","last_peered":"2024-03-19T02:54:57.197656+0000","last_clean":"2024-03-19T02:54:57.197656+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T02:54:57.197656+0000","last_undegraded":"2024-03-19T02:54:57.197656+0000","last_fullsized":"2024-03-19T02:54:57.197656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:23.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380801+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T02:54:57.380801+0000","last_peered":"2024-03-19T02:54:57.380801+0000","last_clean":"2024-03-19T02:54:57.380801+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T02:54:57.380801+0000","last_undegraded":"2024-03-19T02:54:57.380801+0000","last_fullsized":"2024-03-19T02:54:57.380801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084121+0000","last_change":"2024-03-19T02:54:56.201523+0000","last_active":"2024-03-19T02:54:59.084121+0000","last_peered":"2024-03-19T02:54:59.084121+0000","last_clean":"2024-03-19T02:54:59.084121+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T02:54:59.084121+0000","last_undegraded":"2024-03-19T02:54:59.084121+0000","last_fullsized":"2024-03-19T02:54:59.084121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:14.430405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081567+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T02:54:59.081567+0000","last_peered":"2024-03-19T02:54:59.081567+0000","last_clean":"2024-03-19T02:54:59.081567+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T02:54:59.081567+0000","last_undegraded":"2024-03-19T02:54:59.081567+0000","last_fullsized":"2024-03-19T02:54:59.081567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380359+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T02:54:57.380359+0000","last_peered":"2024-03-19T02:54:57.380359+0000","last_clean":"2024-03-19T02:54:57.380359+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T02:54:57.380359+0000","last_undegraded":"2024-03-19T02:54:57.380359+0000","last_fullsized":"2024-03-19T02:54:57.380359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381421+0000","last_change":"2024-03-19T02:54:56.205965+0000","last_active":"2024-03-19T02:54:57.381421+0000","last_peered":"2024-03-19T02:54:57.381421+0000","last_clean":"2024-03-19T02:54:57.381421+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T02:54:57.381421+0000","last_undegraded":"2024-03-19T02:54:57.381421+0000","last_fullsized":"2024-03-19T02:54:57.381421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:01.215544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082556+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T02:54:59.082556+0000","last_peered":"2024-03-19T02:54:59.082556+0000","last_clean":"2024-03-19T02:54:59.082556+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T02:54:59.082556+0000","last_undegraded":"2024-03-19T02:54:59.082556+0000","last_fullsized":"2024-03-19T02:54:59.082556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200860+0000","last_change":"2024-03-19T02:54:56.212562+0000","last_active":"2024-03-19T02:54:57.200860+0000","last_peered":"2024-03-19T02:54:57.200860+0000","last_clean":"2024-03-19T02:54:57.200860+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T02:54:57.200860+0000","last_undegraded":"2024-03-19T02:54:57.200860+0000","last_fullsized":"2024-03-19T02:54:57.200860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:24:03.183734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081713+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T02:54:59.081713+0000","last_peered":"2024-03-19T02:54:59.081713+0000","last_clean":"2024-03-19T02:54:59.081713+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T02:54:59.081713+0000","last_undegraded":"2024-03-19T02:54:59.081713+0000","last_fullsized":"2024-03-19T02:54:59.081713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083157+0000","last_change":"2024-03-19T02:54:56.194559+0000","last_active":"2024-03-19T02:54:59.083157+0000","last_peered":"2024-03-19T02:54:59.083157+0000","last_clean":"2024-03-19T02:54:59.083157+0000","last_became_active":"2024-03-19T02:54:56.194165+0000","last_became_peered":"2024-03-19T02:54:56.194165+0000","last_unstale":"2024-03-19T02:54:59.083157+0000","last_undegraded":"2024-03-19T02:54:59.083157+0000","last_fullsized":"2024-03-19T02:54:59.083157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:30.808511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083191+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T02:54:59.083191+0000","last_peered":"2024-03-19T02:54:59.083191+0000","last_clean":"2024-03-19T02:54:59.083191+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T02:54:59.083191+0000","last_undegraded":"2024-03-19T02:54:59.083191+0000","last_fullsized":"2024-03-19T02:54:59.083191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083968+0000","last_change":"2024-03-19T02:54:56.205673+0000","last_active":"2024-03-19T02:54:59.083968+0000","last_peered":"2024-03-19T02:54:59.083968+0000","last_clean":"2024-03-19T02:54:59.083968+0000","last_became_active":"2024-03-19T02:54:56.203498+0000","last_became_peered":"2024-03-19T02:54:56.203498+0000","last_unstale":"2024-03-19T02:54:59.083968+0000","last_undegraded":"2024-03-19T02:54:59.083968+0000","last_fullsized":"2024-03-19T02:54:59.083968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:54.525175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.182131+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T02:55:01.182131+0000","last_peered":"2024-03-19T02:55:01.182131+0000","last_clean":"2024-03-19T02:55:01.182131+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T02:55:01.182131+0000","last_undegraded":"2024-03-19T02:55:01.182131+0000","last_fullsized":"2024-03-19T02:55:01.182131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.198066+0000","last_change":"2024-03-19T02:54:56.191040+0000","last_active":"2024-03-19T02:54:57.198066+0000","last_peered":"2024-03-19T02:54:57.198066+0000","last_clean":"2024-03-19T02:54:57.198066+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T02:54:57.198066+0000","last_undegraded":"2024-03-19T02:54:57.198066+0000","last_fullsized":"2024-03-19T02:54:57.198066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:17.259448+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T02:55:17.259448+0000","last_peered":"2024-03-19T02:55:17.259448+0000","last_clean":"2024-03-19T02:55:17.259448+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T02:55:17.259448+0000","last_undegraded":"2024-03-19T02:55:17.259448+0000","last_fullsized":"2024-03-19T02:55:17.259448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082069+0000","last_change":"2024-03-19T02:54:56.211642+0000","last_active":"2024-03-19T02:54:59.082069+0000","last_peered":"2024-03-19T02:54:59.082069+0000","last_clean":"2024-03-19T02:54:59.082069+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T02:54:59.082069+0000","last_undegraded":"2024-03-19T02:54:59.082069+0000","last_fullsized":"2024-03-19T02:54:59.082069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:35.088711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.183309+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T02:55:01.183309+0000","last_peered":"2024-03-19T02:55:01.183309+0000","last_clean":"2024-03-19T02:55:01.183309+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T02:55:01.183309+0000","last_undegraded":"2024-03-19T02:55:01.183309+0000","last_fullsized":"2024-03-19T02:55:01.183309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083467+0000","last_change":"2024-03-19T02:54:56.207726+0000","last_active":"2024-03-19T02:54:59.083467+0000","last_peered":"2024-03-19T02:54:59.083467+0000","last_clean":"2024-03-19T02:54:59.083467+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T02:54:59.083467+0000","last_undegraded":"2024-03-19T02:54:59.083467+0000","last_fullsized":"2024-03-19T02:54:59.083467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:02.080123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082876+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T02:54:59.082876+0000","last_peered":"2024-03-19T02:54:59.082876+0000","last_clean":"2024-03-19T02:54:59.082876+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T02:54:59.082876+0000","last_undegraded":"2024-03-19T02:54:59.082876+0000","last_fullsized":"2024-03-19T02:54:59.082876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082189+0000","last_change":"2024-03-19T02:54:56.194724+0000","last_active":"2024-03-19T02:54:59.082189+0000","last_peered":"2024-03-19T02:54:59.082189+0000","last_clean":"2024-03-19T02:54:59.082189+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T02:54:59.082189+0000","last_undegraded":"2024-03-19T02:54:59.082189+0000","last_fullsized":"2024-03-19T02:54:59.082189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:02.129031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081272+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T02:54:59.081272+0000","last_peered":"2024-03-19T02:54:59.081272+0000","last_clean":"2024-03-19T02:54:59.081272+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T02:54:59.081272+0000","last_undegraded":"2024-03-19T02:54:59.081272+0000","last_fullsized":"2024-03-19T02:54:59.081272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081378+0000","last_change":"2024-03-19T02:54:56.199129+0000","last_active":"2024-03-19T02:54:59.081378+0000","last_peered":"2024-03-19T02:54:59.081378+0000","last_clean":"2024-03-19T02:54:59.081378+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T02:54:59.081378+0000","last_undegraded":"2024-03-19T02:54:59.081378+0000","last_fullsized":"2024-03-19T02:54:59.081378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:02.391740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.186421+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T02:55:01.186421+0000","last_peered":"2024-03-19T02:55:01.186421+0000","last_clean":"2024-03-19T02:55:01.186421+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T02:55:01.186421+0000","last_undegraded":"2024-03-19T02:55:01.186421+0000","last_fullsized":"2024-03-19T02:55:01.186421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081716+0000","last_change":"2024-03-19T02:54:56.195336+0000","last_active":"2024-03-19T02:54:59.081716+0000","last_peered":"2024-03-19T02:54:59.081716+0000","last_clean":"2024-03-19T02:54:59.081716+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T02:54:59.081716+0000","last_undegraded":"2024-03-19T02:54:59.081716+0000","last_fullsized":"2024-03-19T02:54:59.081716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:27:55.248863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197959+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T02:54:57.197959+0000","last_peered":"2024-03-19T02:54:57.197959+0000","last_clean":"2024-03-19T02:54:57.197959+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T02:54:57.197959+0000","last_undegraded":"2024-03-19T02:54:57.197959+0000","last_fullsized":"2024-03-19T02:54:57.197959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081981+0000","last_change":"2024-03-19T02:54:56.202826+0000","last_active":"2024-03-19T02:54:59.081981+0000","last_peered":"2024-03-19T02:54:59.081981+0000","last_clean":"2024-03-19T02:54:59.081981+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T02:54:59.081981+0000","last_undegraded":"2024-03-19T02:54:59.081981+0000","last_fullsized":"2024-03-19T02:54:59.081981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:18.910192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188015+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T02:55:01.188015+0000","last_peered":"2024-03-19T02:55:01.188015+0000","last_clean":"2024-03-19T02:55:01.188015+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T02:55:01.188015+0000","last_undegraded":"2024-03-19T02:55:01.188015+0000","last_fullsized":"2024-03-19T02:55:01.188015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082465+0000","last_change":"2024-03-19T02:54:56.197058+0000","last_active":"2024-03-19T02:54:59.082465+0000","last_peered":"2024-03-19T02:54:59.082465+0000","last_clean":"2024-03-19T02:54:59.082465+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T02:54:59.082465+0000","last_undegraded":"2024-03-19T02:54:59.082465+0000","last_fullsized":"2024-03-19T02:54:59.082465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:32:45.977096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201693+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T02:54:57.201693+0000","last_peered":"2024-03-19T02:54:57.201693+0000","last_clean":"2024-03-19T02:54:57.201693+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T02:54:57.201693+0000","last_undegraded":"2024-03-19T02:54:57.201693+0000","last_fullsized":"2024-03-19T02:54:57.201693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083245+0000","last_change":"2024-03-19T02:54:56.195398+0000","last_active":"2024-03-19T02:54:59.083245+0000","last_peered":"2024-03-19T02:54:59.083245+0000","last_clean":"2024-03-19T02:54:59.083245+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T02:54:59.083245+0000","last_undegraded":"2024-03-19T02:54:59.083245+0000","last_fullsized":"2024-03-19T02:54:59.083245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:51.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197580+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T02:54:57.197580+0000","last_peered":"2024-03-19T02:54:57.197580+0000","last_clean":"2024-03-19T02:54:57.197580+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T02:54:57.197580+0000","last_undegraded":"2024-03-19T02:54:57.197580+0000","last_fullsized":"2024-03-19T02:54:57.197580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082198+0000","last_change":"2024-03-19T02:54:56.202566+0000","last_active":"2024-03-19T02:54:59.082198+0000","last_peered":"2024-03-19T02:54:59.082198+0000","last_clean":"2024-03-19T02:54:59.082198+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T02:54:59.082198+0000","last_undegraded":"2024-03-19T02:54:59.082198+0000","last_fullsized":"2024-03-19T02:54:59.082198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:00.526051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081262+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T02:54:59.081262+0000","last_peered":"2024-03-19T02:54:59.081262+0000","last_clean":"2024-03-19T02:54:59.081262+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T02:54:59.081262+0000","last_undegraded":"2024-03-19T02:54:59.081262+0000","last_fullsized":"2024-03-19T02:54:59.081262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082720+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T02:54:59.082720+0000","last_peered":"2024-03-19T02:54:59.082720+0000","last_clean":"2024-03-19T02:54:59.082720+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T02:54:59.082720+0000","last_undegraded":"2024-03-19T02:54:59.082720+0000","last_fullsized":"2024-03-19T02:54:59.082720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380926+0000","last_change":"2024-03-19T02:54:56.205069+0000","last_active":"2024-03-19T02:54:57.380926+0000","last_peered":"2024-03-19T02:54:57.380926+0000","last_clean":"2024-03-19T02:54:57.380926+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T02:54:57.380926+0000","last_undegraded":"2024-03-19T02:54:57.380926+0000","last_fullsized":"2024-03-19T02:54:57.380926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:04.267451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081136+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T02:54:59.081136+0000","last_peered":"2024-03-19T02:54:59.081136+0000","last_clean":"2024-03-19T02:54:59.081136+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T02:54:59.081136+0000","last_undegraded":"2024-03-19T02:54:59.081136+0000","last_fullsized":"2024-03-19T02:54:59.081136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083127+0000","last_change":"2024-03-19T02:54:56.198564+0000","last_active":"2024-03-19T02:54:59.083127+0000","last_peered":"2024-03-19T02:54:59.083127+0000","last_clean":"2024-03-19T02:54:59.083127+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T02:54:59.083127+0000","last_undegraded":"2024-03-19T02:54:59.083127+0000","last_fullsized":"2024-03-19T02:54:59.083127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.196909+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T02:54:57.196909+0000","last_peered":"2024-03-19T02:54:57.196909+0000","last_clean":"2024-03-19T02:54:57.196909+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T02:54:57.196909+0000","last_undegraded":"2024-03-19T02:54:57.196909+0000","last_fullsized":"2024-03-19T02:54:57.196909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200870+0000","last_change":"2024-03-19T02:54:56.207369+0000","last_active":"2024-03-19T02:54:57.200870+0000","last_peered":"2024-03-19T02:54:57.200870+0000","last_clean":"2024-03-19T02:54:57.200870+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T02:54:57.200870+0000","last_undegraded":"2024-03-19T02:54:57.200870+0000","last_fullsized":"2024-03-19T02:54:57.200870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:07.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083032+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T02:54:59.083032+0000","last_peered":"2024-03-19T02:54:59.083032+0000","last_clean":"2024-03-19T02:54:59.083032+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T02:54:59.083032+0000","last_undegraded":"2024-03-19T02:54:59.083032+0000","last_fullsized":"2024-03-19T02:54:59.083032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197915+0000","last_change":"2024-03-19T02:54:56.215948+0000","last_active":"2024-03-19T02:54:57.197915+0000","last_peered":"2024-03-19T02:54:57.197915+0000","last_clean":"2024-03-19T02:54:57.197915+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T02:54:57.197915+0000","last_undegraded":"2024-03-19T02:54:57.197915+0000","last_fullsized":"2024-03-19T02:54:57.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:04.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201270+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T02:54:57.201270+0000","last_peered":"2024-03-19T02:54:57.201270+0000","last_clean":"2024-03-19T02:54:57.201270+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T02:54:57.201270+0000","last_undegraded":"2024-03-19T02:54:57.201270+0000","last_fullsized":"2024-03-19T02:54:57.201270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083284+0000","last_change":"2024-03-19T02:54:56.204148+0000","last_active":"2024-03-19T02:54:59.083284+0000","last_peered":"2024-03-19T02:54:59.083284+0000","last_clean":"2024-03-19T02:54:59.083284+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T02:54:59.083284+0000","last_undegraded":"2024-03-19T02:54:59.083284+0000","last_fullsized":"2024-03-19T02:54:59.083284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:45.067404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082361+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T02:54:59.082361+0000","last_peered":"2024-03-19T02:54:59.082361+0000","last_clean":"2024-03-19T02:54:59.082361+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T02:54:59.082361+0000","last_undegraded":"2024-03-19T02:54:59.082361+0000","last_fullsized":"2024-03-19T02:54:59.082361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201353+0000","last_change":"2024-03-19T02:54:56.206499+0000","last_active":"2024-03-19T02:54:57.201353+0000","last_peered":"2024-03-19T02:54:57.201353+0000","last_clean":"2024-03-19T02:54:57.201353+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T02:54:57.201353+0000","last_undegraded":"2024-03-19T02:54:57.201353+0000","last_fullsized":"2024-03-19T02:54:57.201353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:24.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201786+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T02:54:57.201786+0000","last_peered":"2024-03-19T02:54:57.201786+0000","last_clean":"2024-03-19T02:54:57.201786+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T02:54:57.201786+0000","last_undegraded":"2024-03-19T02:54:57.201786+0000","last_fullsized":"2024-03-19T02:54:57.201786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381435+0000","last_change":"2024-03-19T02:54:56.199541+0000","last_active":"2024-03-19T02:54:57.381435+0000","last_peered":"2024-03-19T02:54:57.381435+0000","last_clean":"2024-03-19T02:54:57.381435+0000","last_became_active":"2024-03-19T02:54:56.199251+0000","last_became_peered":"2024-03-19T02:54:56.199251+0000","last_unstale":"2024-03-19T02:54:57.381435+0000","last_undegraded":"2024-03-19T02:54:57.381435+0000","last_fullsized":"2024-03-19T02:54:57.381435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:32.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177520+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T02:55:01.177520+0000","last_peered":"2024-03-19T02:55:01.177520+0000","last_clean":"2024-03-19T02:55:01.177520+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T02:55:01.177520+0000","last_undegraded":"2024-03-19T02:55:01.177520+0000","last_fullsized":"2024-03-19T02:55:01.177520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083860+0000","last_change":"2024-03-19T02:54:56.209793+0000","last_active":"2024-03-19T02:54:59.083860+0000","last_peered":"2024-03-19T02:54:59.083860+0000","last_clean":"2024-03-19T02:54:59.083860+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T02:54:59.083860+0000","last_undegraded":"2024-03-19T02:54:59.083860+0000","last_fullsized":"2024-03-19T02:54:59.083860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:24.506221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081183+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T02:54:59.081183+0000","last_peered":"2024-03-19T02:54:59.081183+0000","last_clean":"2024-03-19T02:54:59.081183+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T02:54:59.081183+0000","last_undegraded":"2024-03-19T02:54:59.081183+0000","last_fullsized":"2024-03-19T02:54:59.081183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.197947+0000","last_change":"2024-03-19T02:54:56.192551+0000","last_active":"2024-03-19T02:54:57.197947+0000","last_peered":"2024-03-19T02:54:57.197947+0000","last_clean":"2024-03-19T02:54:57.197947+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T02:54:57.197947+0000","last_undegraded":"2024-03-19T02:54:57.197947+0000","last_fullsized":"2024-03-19T02:54:57.197947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:12.506391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.200943+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T02:54:57.200943+0000","last_peered":"2024-03-19T02:54:57.200943+0000","last_clean":"2024-03-19T02:54:57.200943+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T02:54:57.200943+0000","last_undegraded":"2024-03-19T02:54:57.200943+0000","last_fullsized":"2024-03-19T02:54:57.200943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380791+0000","last_change":"2024-03-19T02:54:56.200540+0000","last_active":"2024-03-19T02:54:57.380791+0000","last_peered":"2024-03-19T02:54:57.380791+0000","last_clean":"2024-03-19T02:54:57.380791+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T02:54:57.380791+0000","last_undegraded":"2024-03-19T02:54:57.380791+0000","last_fullsized":"2024-03-19T02:54:57.380791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:16.056937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.201144+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T02:54:57.201144+0000","last_peered":"2024-03-19T02:54:57.201144+0000","last_clean":"2024-03-19T02:54:57.201144+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T02:54:57.201144+0000","last_undegraded":"2024-03-19T02:54:57.201144+0000","last_fullsized":"2024-03-19T02:54:57.201144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.380685+0000","last_change":"2024-03-19T02:54:56.216275+0000","last_active":"2024-03-19T02:54:57.380685+0000","last_peered":"2024-03-19T02:54:57.380685+0000","last_clean":"2024-03-19T02:54:57.380685+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T02:54:57.380685+0000","last_undegraded":"2024-03-19T02:54:57.380685+0000","last_fullsized":"2024-03-19T02:54:57.380685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:36.005624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177763+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T02:55:01.177763+0000","last_peered":"2024-03-19T02:55:01.177763+0000","last_clean":"2024-03-19T02:55:01.177763+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T02:55:01.177763+0000","last_undegraded":"2024-03-19T02:55:01.177763+0000","last_fullsized":"2024-03-19T02:55:01.177763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.081381+0000","last_change":"2024-03-19T02:54:56.193869+0000","last_active":"2024-03-19T02:54:59.081381+0000","last_peered":"2024-03-19T02:54:59.081381+0000","last_clean":"2024-03-19T02:54:59.081381+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T02:54:59.081381+0000","last_undegraded":"2024-03-19T02:54:59.081381+0000","last_fullsized":"2024-03-19T02:54:59.081381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:48.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.083313+0000","last_change":"2024-03-19T02:54:56.184712+0000","last_active":"2024-03-19T02:54:59.083313+0000","last_peered":"2024-03-19T02:54:59.083313+0000","last_clean":"2024-03-19T02:54:59.083313+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T02:54:59.083313+0000","last_undegraded":"2024-03-19T02:54:59.083313+0000","last_fullsized":"2024-03-19T02:54:59.083313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:43.034216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.177777+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T02:55:01.177777+0000","last_peered":"2024-03-19T02:55:01.177777+0000","last_clean":"2024-03-19T02:55:01.177777+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T02:55:01.177777+0000","last_undegraded":"2024-03-19T02:55:01.177777+0000","last_fullsized":"2024-03-19T02:55:01.177777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.082976+0000","last_change":"2024-03-19T02:54:56.182982+0000","last_active":"2024-03-19T02:54:59.082976+0000","last_peered":"2024-03-19T02:54:59.082976+0000","last_clean":"2024-03-19T02:54:59.082976+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T02:54:59.082976+0000","last_undegraded":"2024-03-19T02:54:59.082976+0000","last_fullsized":"2024-03-19T02:54:59.082976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:42:00.237706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.178273+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T02:55:01.178273+0000","last_peered":"2024-03-19T02:55:01.178273+0000","last_clean":"2024-03-19T02:55:01.178273+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T02:55:01.178273+0000","last_undegraded":"2024-03-19T02:55:01.178273+0000","last_fullsized":"2024-03-19T02:55:01.178273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:59.084001+0000","last_change":"2024-03-19T02:54:56.206289+0000","last_active":"2024-03-19T02:54:59.084001+0000","last_peered":"2024-03-19T02:54:59.084001+0000","last_clean":"2024-03-19T02:54:59.084001+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T02:54:59.084001+0000","last_undegraded":"2024-03-19T02:54:59.084001+0000","last_fullsized":"2024-03-19T02:54:59.084001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:28.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381552+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T02:54:57.381552+0000","last_peered":"2024-03-19T02:54:57.381552+0000","last_clean":"2024-03-19T02:54:57.381552+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T02:54:57.381552+0000","last_undegraded":"2024-03-19T02:54:57.381552+0000","last_fullsized":"2024-03-19T02:54:57.381552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:54:57.381442+0000","last_change":"2024-03-19T02:54:56.192900+0000","last_active":"2024-03-19T02:54:57.381442+0000","last_peered":"2024-03-19T02:54:57.381442+0000","last_clean":"2024-03-19T02:54:57.381442+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T02:54:57.381442+0000","last_undegraded":"2024-03-19T02:54:57.381442+0000","last_fullsized":"2024-03-19T02:54:57.381442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:32.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T02:55:01.188465+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T02:55:01.188465+0000","last_peered":"2024-03-19T02:55:01.188465+0000","last_clean":"2024-03-19T02:55:01.188465+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T02:55:01.188465+0000","last_undegraded":"2024-03-19T02:55:01.188465+0000","last_fullsized":"2024-03-19T02:55:01.188465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2664,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661884,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":661940,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:55:24.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:24.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.763+0000 7f9d59a00640 1 Processor -- start 2024-03-19T02:55:24.764 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.763+0000 7f9d59a00640 1 -- start start 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.764+0000 7f9d59a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9d54154dc0 0x7f9d54071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.764+0000 7f9d59a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5415b740 con 0x7f9d540c60d0 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d59a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5415ba20 con 0x7f9d54072380 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d59a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f9d540c8400 con 0x7f9d54154dc0 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d4a000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9d540c60d0 0x7f9d54071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46242/0 (socket says 172.21.15.92:46242) 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d53400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9d54072380 0x7f9d540c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:49384/0 (socket says 172.21.15.92:49384) 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d4a000640 1 -- 172.21.15.92:0/128421300 learned_addr learned my addr 172.21.15.92:0/128421300 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d52a00640 1 --2- 172.21.15.92:0/128421300 >> v2:172.21.15.159:3300/0 conn(0x7f9d54154dc0 0x7f9d54071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063666213 0 0) 0x7f9d5415ba20 con 0x7f9d54072380 2024-03-19T02:55:24.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d38003520 con 0x7f9d54072380 2024-03-19T02:55:24.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541853201 0 0) 0x7f9d5415b740 con 0x7f9d540c60d0 2024-03-19T02:55:24.767 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.765+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d5415ba20 con 0x7f9d540c60d0 2024-03-19T02:55:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.766+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2532637941 0 0) 0x7f9d38003520 con 0x7f9d54072380 2024-03-19T02:55:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.766+0000 7f9d52a00640 1 -- 172.21.15.92:0/128421300 >> v1:172.21.15.159:6789/0 conn(0x7f9d54072380 legacy=0x7f9d540c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.766+0000 7f9d52a00640 1 -- 172.21.15.92:0/128421300 >> v1:172.21.15.92:6789/0 conn(0x7f9d540c60d0 legacy=0x7f9d54071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.766+0000 7f9d52a00640 1 -- 172.21.15.92:0/128421300 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d540c86e0 con 0x7f9d54154dc0 2024-03-19T02:55:24.768 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.767+0000 7f9d52a00640 1 --2- 172.21.15.92:0/128421300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d54154dc0 0x7f9d54071190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9d440097a0 tx=0x7f9d4402b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a37a3668cb2de895 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44037070 con 0x7f9d54154dc0 2024-03-19T02:55:24.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44030510 con 0x7f9d54154dc0 2024-03-19T02:55:24.769 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d52000640 1 -- 172.21.15.92:0/128421300 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0015f0 con 0x7f9d54154dc0 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d59a00640 1 -- 172.21.15.92:0/128421300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d54154dc0 msgr2=0x7f9d54071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d59a00640 1 --2- 172.21.15.92:0/128421300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d54154dc0 0x7f9d54071190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9d440097a0 tx=0x7f9d4402b680 comp rx=0 tx=0).stop 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d59a00640 1 -- 172.21.15.92:0/128421300 shutdown_connections 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d59a00640 1 --2- 172.21.15.92:0/128421300 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d54154dc0 0x7f9d54071190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.768+0000 7f9d59a00640 1 -- 172.21.15.92:0/128421300 >> 172.21.15.92:0/128421300 conn(0x7f9d540c8d90 msgr2=0x7f9d540c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.769+0000 7f9d59a00640 1 -- 172.21.15.92:0/128421300 shutdown_connections 2024-03-19T02:55:24.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.769+0000 7f9d59a00640 1 -- 172.21.15.92:0/128421300 wait complete. 2024-03-19T02:55:24.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.770+0000 7f9d59a00640 1 Processor -- start 2024-03-19T02:55:24.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.770+0000 7f9d59a00640 1 -- start start 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 0x7f9d540cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d54154dc0 0x7f9d541f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9d540c7520 con 0x7f9d54072380 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9d540c73a0 con 0x7f9d540c60d0 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.771+0000 7f9d59a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9d540c76a0 con 0x7f9d54154dc0 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d53400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d54154dc0 0x7f9d541f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d4a000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 0x7f9d540cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49596/0 (socket says 172.21.15.92:49596) 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 -- 172.21.15.92:0/3635930712 learned_addr learned my addr 172.21.15.92:0/3635930712 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:24.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 -- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d54154dc0 msgr2=0x7f9d541f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d54154dc0 0x7f9d541f2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 -- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 msgr2=0x7f9d540cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 0x7f9d540cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 -- 172.21.15.92:0/3635930712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5415c440 con 0x7f9d54072380 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d4a000640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 0x7f9d540cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.772+0000 7f9d52a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d440098c0 tx=0x7f9d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.773+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44041070 con 0x7f9d54072380 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.773+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4403c870 con 0x7f9d54072380 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.773+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44037040 con 0x7f9d54072380 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.773+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5415c6e0 con 0x7f9d54072380 2024-03-19T02:55:24.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.773+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d5415cca0 con 0x7f9d54072380 2024-03-19T02:55:24.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.774+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d54068de0 con 0x7f9d54072380 2024-03-19T02:55:24.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.775+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d44044070 con 0x7f9d54072380 2024-03-19T02:55:24.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.776+0000 7f9d4b400640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 0x7f9d3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:24.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.776+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9d440ac460 con 0x7f9d54072380 2024-03-19T02:55:24.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.776+0000 7f9d4a000640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 0x7f9d3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:24.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.777+0000 7f9d4a000640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 0x7f9d3006e090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d541eb430 tx=0x7f9d3c001a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:24.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.781+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d54068de0 con 0x7f9d54072380 2024-03-19T02:55:24.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.984+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d54158480 con 0x7f9d54072380 2024-03-19T02:55:24.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.986+0000 7f9d4b400640 1 -- 172.21.15.92:0/3635930712 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f9d4407a7d0 con 0x7f9d54072380 2024-03-19T02:55:24.988 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:24.988 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:24.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.989+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 msgr2=0x7f9d3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.989+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 0x7f9d3006e090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d541eb430 tx=0x7f9d3c001a20 comp rx=0 tx=0).stop 2024-03-19T02:55:24.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.989+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 msgr2=0x7f9d541f0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:24.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.989+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d440098c0 tx=0x7f9d440046c0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 shutdown_connections 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d3006bbd0 0x7f9d3006e090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d54154dc0 0x7f9d541f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d540c60d0 0x7f9d540cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 --2- 172.21.15.92:0/3635930712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54072380 0x7f9d541f0af0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 >> 172.21.15.92:0/3635930712 conn(0x7f9d540c8d90 msgr2=0x7f9d54152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 shutdown_connections 2024-03-19T02:55:24.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:24.990+0000 7f9d59a00640 1 -- 172.21.15.92:0/3635930712 wait complete. 2024-03-19T02:55:25.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:55:25.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:55:25.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.217+0000 7f6bc1400640 1 Processor -- start 2024-03-19T02:55:25.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.217+0000 7f6bc1400640 1 -- start start 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bc1400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bc1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bc1400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bbc0c3b80 con 0x7f6bbc151900 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bc1400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6bbc158860 con 0x7f6bbc151ce0 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bc1400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6bbc072e70 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bbb200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.218+0000 7f6bb9e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bba800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6bbc151900 0x7f6bbc0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:46252/0 (socket says 172.21.15.92:46252) 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bbb200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41788/0 (socket says 172.21.15.92:41788) 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bba800640 1 -- 172.21.15.92:0/356230576 learned_addr learned my addr 172.21.15.92:0/356230576 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bb9e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46200/0 (socket says 172.21.15.92:46200) 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bbb200640 1 -- 172.21.15.92:0/356230576 >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 msgr2=0x7f6bbc158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bbb200640 1 --2- 172.21.15.92:0/356230576 >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bbb200640 1 -- 172.21.15.92:0/356230576 >> v1:172.21.15.92:6789/0 conn(0x7f6bbc151900 legacy=0x7f6bbc0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bbb200640 1 -- 172.21.15.92:0/356230576 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bbc073150 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.219+0000 7f6bb9e00640 1 --2- 172.21.15.92:0/356230576 >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.220+0000 7f6bbb200640 1 --2- 172.21.15.92:0/356230576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009d90 tx=0x7f6bb402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f42f038ca82fef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.220+0000 7f6bb9400640 1 -- 172.21.15.92:0/356230576 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4039070 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.220+0000 7f6bb9400640 1 -- 172.21.15.92:0/356230576 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bb402cce0 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.220+0000 7f6bb9400640 1 -- 172.21.15.92:0/356230576 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4034660 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 -- 172.21.15.92:0/356230576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 msgr2=0x7f6bbc0c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 --2- 172.21.15.92:0/356230576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009d90 tx=0x7f6bb402bf30 comp rx=0 tx=0).stop 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 -- 172.21.15.92:0/356230576 shutdown_connections 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 --2- 172.21.15.92:0/356230576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc0c7da0 0x7f6bbc0c8230 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 --2- 172.21.15.92:0/356230576 >> v2:172.21.15.159:3300/0 conn(0x7f6bbc151ce0 0x7f6bbc158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 -- 172.21.15.92:0/356230576 >> 172.21.15.92:0/356230576 conn(0x7f6bbc0c8d90 msgr2=0x7f6bbc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:25.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 -- 172.21.15.92:0/356230576 shutdown_connections 2024-03-19T02:55:25.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.221+0000 7f6bc1400640 1 -- 172.21.15.92:0/356230576 wait complete. 2024-03-19T02:55:25.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.222+0000 7f6bc1400640 1 Processor -- start 2024-03-19T02:55:25.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 -- start start 2024-03-19T02:55:25.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bbc0c7da0 0x7f6bbc1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 0x7f6bbc146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bbc0c9af0 con 0x7f6bbc0c7da0 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6bbc071eb0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.223+0000 7f6bc1400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6bbc0c9c70 con 0x7f6bbc151900 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bb9e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 0x7f6bbc146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bba800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bbc0c7da0 0x7f6bbc1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bb9e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 0x7f6bbc146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41796/0 (socket says 172.21.15.92:41796) 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46216/0 (socket says 172.21.15.92:46216) 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bb9e00640 1 -- 172.21.15.92:0/1761199531 learned_addr learned my addr 172.21.15.92:0/1761199531 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 -- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 msgr2=0x7f6bbc146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 0x7f6bbc146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 -- 172.21.15.92:0/1761199531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bbc0c7da0 msgr2=0x7f6bbc1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bbc0c7da0 0x7f6bbc1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 -- 172.21.15.92:0/1761199531 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bbc1525c0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bbb200640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4033000 tx=0x7f6bb4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.224+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4039050 con 0x7f6bbc151ce0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.225+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bb4045070 con 0x7f6bbc151ce0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.225+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bbc153bf0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.225+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4009450 con 0x7f6bbc151ce0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.225+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bbc1541b0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.226+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bbc0684c0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.227+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6bb403e850 con 0x7f6bbc151ce0 2024-03-19T02:55:25.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.227+0000 7f6bb2a00640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 0x7f6b9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.227+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6bb40ad7a0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.228+0000 7f6bba800640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 0x7f6b9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.228+0000 7f6bba800640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 0x7f6b9c06e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bac007fd0 tx=0x7f6bac007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.232+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6bb407bcc0 con 0x7f6bbc151ce0 2024-03-19T02:55:25.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.413+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6bbc152a10 con 0x7f6bbc151ce0 2024-03-19T02:55:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.414+0000 7f6bb2a00640 1 -- 172.21.15.92:0/1761199531 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f6bb407fd60 con 0x7f6bbc151ce0 2024-03-19T02:55:25.415 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 msgr2=0x7f6b9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 0x7f6b9c06e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bac007fd0 tx=0x7f6bac007e80 comp rx=0 tx=0).stop 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 msgr2=0x7f6bbc149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4033000 tx=0x7f6bb4004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 shutdown_connections 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b9c06bbd0 0x7f6b9c06e090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bbc151ce0 0x7f6bbc149c60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bbc151900 0x7f6bbc146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 --2- 172.21.15.92:0/1761199531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bbc0c7da0 0x7f6bbc1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.416+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 >> 172.21.15.92:0/1761199531 conn(0x7f6bbc0c8d90 msgr2=0x7f6bbc156670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.417+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 shutdown_connections 2024-03-19T02:55:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.417+0000 7f6bc1400640 1 -- 172.21.15.92:0/1761199531 wait complete. 2024-03-19T02:55:25.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:55:25.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.648+0000 7f80a2800640 1 Processor -- start 2024-03-19T02:55:25.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.648+0000 7f80a2800640 1 -- start start 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f809c0d0660 0x7f809c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f809c0cb9f0 0x7f809c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f809c0c79d0 con 0x7f809c0d0660 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f809c0ccf40 con 0x7f809c0cb9f0 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a2800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f809c0cdd10 con 0x7f809c0d0a40 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:41804/0 (socket says 172.21.15.92:41804) 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a0a00640 1 -- 172.21.15.92:0/2173104980 learned_addr learned my addr 172.21.15.92:0/2173104980 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a1400640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.92:3300/0 conn(0x7f809c0d0660 0x7f809c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.650+0000 7f80a1e00640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.159:3300/0 conn(0x7f809c0cb9f0 0x7f809c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.651+0000 7f80a0a00640 1 -- 172.21.15.92:0/2173104980 >> v2:172.21.15.159:3300/0 conn(0x7f809c0cb9f0 msgr2=0x7f809c0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:25.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.651+0000 7f80a0a00640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.159:3300/0 conn(0x7f809c0cb9f0 0x7f809c0c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.651+0000 7f80a0a00640 1 -- 172.21.15.92:0/2173104980 >> v2:172.21.15.92:3300/0 conn(0x7f809c0d0660 msgr2=0x7f809c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:25.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.651+0000 7f80a0a00640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.92:3300/0 conn(0x7f809c0d0660 0x7f809c0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.651+0000 7f80a0a00640 1 -- 172.21.15.92:0/2173104980 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c0cd190 con 0x7f809c0d0a40 2024-03-19T02:55:25.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.652+0000 7f80a0a00640 1 --2- 172.21.15.92:0/2173104980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f809c0b33f0 tx=0x7f808c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690089c3a0a890f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.652+0000 7f809be00640 1 -- 172.21.15.92:0/2173104980 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c02ca80 con 0x7f809c0d0a40 2024-03-19T02:55:25.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.652+0000 7f809be00640 1 -- 172.21.15.92:0/2173104980 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c02cc20 con 0x7f809c0d0a40 2024-03-19T02:55:25.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.652+0000 7f809be00640 1 -- 172.21.15.92:0/2173104980 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0343f0 con 0x7f809c0d0a40 2024-03-19T02:55:25.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 -- 172.21.15.92:0/2173104980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 msgr2=0x7f809c0cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 --2- 172.21.15.92:0/2173104980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f809c0b33f0 tx=0x7f808c02bee0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 -- 172.21.15.92:0/2173104980 shutdown_connections 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.159:3300/0 conn(0x7f809c0cb9f0 0x7f809c0c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 --2- 172.21.15.92:0/2173104980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0d0a40 0x7f809c0cb4b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 --2- 172.21.15.92:0/2173104980 >> v2:172.21.15.92:3300/0 conn(0x7f809c0d0660 0x7f809c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 -- 172.21.15.92:0/2173104980 >> 172.21.15.92:0/2173104980 conn(0x7f809c072b50 msgr2=0x7f809c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 -- 172.21.15.92:0/2173104980 shutdown_connections 2024-03-19T02:55:25.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.653+0000 7f80a2800640 1 -- 172.21.15.92:0/2173104980 wait complete. 2024-03-19T02:55:25.656 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.655+0000 7f80a2800640 1 Processor -- start 2024-03-19T02:55:25.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.655+0000 7f80a2800640 1 -- start start 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0cb9f0 0x7f809c1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 0x7f809c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f809c0cda30 con 0x7f809c0d0660 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f809c0cd750 con 0x7f809c0d0a40 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.656+0000 7f80a2800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f809c0c83a0 con 0x7f809c0cb9f0 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46220/0 (socket says 172.21.15.92:46220) 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 -- 172.21.15.92:0/2287758567 learned_addr learned my addr 172.21.15.92:0/2287758567 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1400640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0cb9f0 0x7f809c1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a0a00640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 0x7f809c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 -- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0cb9f0 msgr2=0x7f809c1f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0cb9f0 0x7f809c1f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 -- 172.21.15.92:0/2287758567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 msgr2=0x7f809c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 0x7f809c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 -- 172.21.15.92:0/2287758567 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c1f0790 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a0a00640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 0x7f809c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.657+0000 7f80a1e00640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f809400e7a0 tx=0x7f809400ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.658+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094017070 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.658+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f809400c940 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.658+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809400cd30 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.658+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809c1f0a30 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.658+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809c1f0ff0 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.659+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809c04f580 con 0x7f809c0d0a40 2024-03-19T02:55:25.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.660+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f809401a070 con 0x7f809c0d0a40 2024-03-19T02:55:25.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.661+0000 7f8099600640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 0x7f808006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:25.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.661+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f809408a9c0 con 0x7f809c0d0a40 2024-03-19T02:55:25.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.661+0000 7f80a1400640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 0x7f808006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:25.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.662+0000 7f80a1400640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 0x7f808006e0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8090009b40 tx=0x7f8090009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:25.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.665+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8094058db0 con 0x7f809c0d0a40 2024-03-19T02:55:25.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.864+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f809c14e020 con 0x7f809c0d0a40 2024-03-19T02:55:25.866 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.865+0000 7f8099600640 1 -- 172.21.15.92:0/2287758567 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f809405ce50 con 0x7f809c0d0a40 2024-03-19T02:55:25.866 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 msgr2=0x7f808006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 0x7f808006e0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8090009b40 tx=0x7f8090009190 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 msgr2=0x7f809c14f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f809400e7a0 tx=0x7f809400ec60 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 shutdown_connections 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f808006bc20 0x7f808006e0e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f809c0d0a40 0x7f809c14f5d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0d0660 0x7f809c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 --2- 172.21.15.92:0/2287758567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f809c0cb9f0 0x7f809c1f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:25.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.867+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 >> 172.21.15.92:0/2287758567 conn(0x7f809c072b50 msgr2=0x7f809c151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:25.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.868+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 shutdown_connections 2024-03-19T02:55:25.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:25.868+0000 7f80a2800640 1 -- 172.21.15.92:0/2287758567 wait complete. 2024-03-19T02:55:25.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:55:26.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.102+0000 7f3ba6a00640 1 Processor -- start 2024-03-19T02:55:26.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.102+0000 7f3ba6a00640 1 -- start start 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba6a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 0x7f3ba00cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba6a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ba00cdd10 con 0x7f3ba00d0a40 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba6a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ba00cd190 con 0x7f3ba00d0660 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba6a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba00c79d0 con 0x7f3ba00c8570 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba5600640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3ba00d0660 0x7f3ba00cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:41632/0 (socket says 172.21.15.92:41632) 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba5600640 1 -- 172.21.15.92:0/155906017 learned_addr learned my addr 172.21.15.92:0/155906017 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:26.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3ba6000640 1 --2- 172.21.15.92:0/155906017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 0x7f3ba00cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.104+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770420199 0 0) 0x7f3ba00cd190 con 0x7f3ba00d0660 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.105+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b88003520 con 0x7f3ba00d0660 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.105+0000 7f3ba6000640 1 -- 172.21.15.92:0/155906017 >> v1:172.21.15.159:6789/0 conn(0x7f3ba00d0660 legacy=0x7f3ba00cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.105+0000 7f3ba6000640 1 -- 172.21.15.92:0/155906017 >> v1:172.21.15.92:6789/0 conn(0x7f3ba00d0a40 legacy=0x7f3ba00c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.105+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38616899 0 0) 0x7f3ba00cdd10 con 0x7f3ba00d0a40 2024-03-19T02:55:26.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.105+0000 7f3ba6000640 1 -- 172.21.15.92:0/155906017 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba00cd470 con 0x7f3ba00c8570 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.106+0000 7f3ba6000640 1 --2- 172.21.15.92:0/155906017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 0x7f3ba00cca00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009dc0 tx=0x7f3b9802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc4044c94dcad3a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.106+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98039070 con 0x7f3ba00c8570 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.106+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b9802cd50 con 0x7f3ba00c8570 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.106+0000 7f3b9fe00640 1 -- 172.21.15.92:0/155906017 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98034660 con 0x7f3ba00c8570 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 -- 172.21.15.92:0/155906017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 msgr2=0x7f3ba00cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/155906017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 0x7f3ba00cca00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009dc0 tx=0x7f3b9802bfa0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 -- 172.21.15.92:0/155906017 shutdown_connections 2024-03-19T02:55:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/155906017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00c8570 0x7f3ba00cca00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 -- 172.21.15.92:0/155906017 >> 172.21.15.92:0/155906017 conn(0x7f3ba0072b50 msgr2=0x7f3ba006d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 -- 172.21.15.92:0/155906017 shutdown_connections 2024-03-19T02:55:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.107+0000 7f3ba6a00640 1 -- 172.21.15.92:0/155906017 wait complete. 2024-03-19T02:55:26.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.109+0000 7f3ba6a00640 1 Processor -- start 2024-03-19T02:55:26.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.109+0000 7f3ba6a00640 1 -- start start 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 0x7f3ba015be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 0x7f3ba0152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba00cda30 con 0x7f3ba00c8570 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba00ccf40 con 0x7f3ba00d0a40 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba00cbbc0 con 0x7f3ba00d0660 2024-03-19T02:55:26.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 0x7f3ba015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 0x7f3ba0152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:40196/0 (socket says 172.21.15.92:40196) 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba6000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 0x7f3ba0152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:42198/0 (socket says 172.21.15.92:42198) 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 -- 172.21.15.92:0/3251898781 learned_addr learned my addr 172.21.15.92:0/3251898781 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 0x7f3ba015be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55546/0 (socket says 172.21.15.92:55546) 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 -- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 msgr2=0x7f3ba0152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 0x7f3ba0152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 -- 172.21.15.92:0/3251898781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 msgr2=0x7f3ba015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 0x7f3ba015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.110+0000 7f3ba4c00640 1 -- 172.21.15.92:0/3251898781 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba0158b60 con 0x7f3ba00d0660 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3ba4c00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c3f0 tx=0x7f3b8c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c017070 con 0x7f3ba00d0660 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b8c0035f0 con 0x7f3ba00d0660 2024-03-19T02:55:26.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c0039e0 con 0x7f3ba00d0660 2024-03-19T02:55:26.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba0158de0 con 0x7f3ba00d0660 2024-03-19T02:55:26.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.111+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba01593a0 con 0x7f3ba00d0660 2024-03-19T02:55:26.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.112+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba004f580 con 0x7f3ba00d0660 2024-03-19T02:55:26.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.113+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3b8c01a070 con 0x7f3ba00d0660 2024-03-19T02:55:26.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.114+0000 7f3b9d600640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 0x7f3b8006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.114+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3b8c089d40 con 0x7f3ba00d0660 2024-03-19T02:55:26.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.114+0000 7f3ba5600640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 0x7f3b8006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.115+0000 7f3ba5600640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 0x7f3b8006e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3b940011e0 tx=0x7f3b94007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.118+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3ba0158b60 con 0x7f3ba00d0660 2024-03-19T02:55:26.315 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.314+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ba00cbe20 con 0x7f3ba00d0660 2024-03-19T02:55:26.316 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.315+0000 7f3b9d600640 1 -- 172.21.15.92:0/3251898781 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f3b8c0581d0 con 0x7f3ba00d0660 2024-03-19T02:55:26.316 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 msgr2=0x7f3b8006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 0x7f3b8006e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3b940011e0 tx=0x7f3b94007d30 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 msgr2=0x7f3ba015ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c3f0 tx=0x7f3b8c00c8b0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 shutdown_connections 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3b8006bc00 0x7f3b8006e0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba00d0a40 0x7f3ba0152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba00d0660 0x7f3ba015ec50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 --2- 172.21.15.92:0/3251898781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba00c8570 0x7f3ba015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 >> 172.21.15.92:0/3251898781 conn(0x7f3ba0072b50 msgr2=0x7f3ba00cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 shutdown_connections 2024-03-19T02:55:26.319 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.317+0000 7f3ba6a00640 1 -- 172.21.15.92:0/3251898781 wait complete. 2024-03-19T02:55:26.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:26.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.541+0000 7f7cdca00640 1 Processor -- start 2024-03-19T02:55:26.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.541+0000 7f7cdca00640 1 -- start start 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cdca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7cd8151900 0x7f7cd8158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cdca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 0x7f7cd80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cdca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cd80c3b80 con 0x7f7cd8151ce0 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cdca00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7cd8158860 con 0x7f7cd8151900 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cdca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd8072e70 con 0x7f7cd80c7da0 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.543+0000 7f7cd6800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 0x7f7cd80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd5400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7cd8151ce0 0x7f7cd80723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:47986/0 (socket says 172.21.15.92:47986) 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd5400640 1 -- 172.21.15.92:0/2328938581 learned_addr learned my addr 172.21.15.92:0/2328938581 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd5e00640 1 --2- 172.21.15.92:0/2328938581 >> v2:172.21.15.92:3300/0 conn(0x7f7cd8151900 0x7f7cd8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd6800640 1 -- 172.21.15.92:0/2328938581 >> v2:172.21.15.92:3300/0 conn(0x7f7cd8151900 msgr2=0x7f7cd8158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd6800640 1 --2- 172.21.15.92:0/2328938581 >> v2:172.21.15.92:3300/0 conn(0x7f7cd8151900 0x7f7cd8158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd6800640 1 -- 172.21.15.92:0/2328938581 >> v1:172.21.15.92:6789/0 conn(0x7f7cd8151ce0 legacy=0x7f7cd80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.544+0000 7f7cd6800640 1 -- 172.21.15.92:0/2328938581 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd8073150 con 0x7f7cd80c7da0 2024-03-19T02:55:26.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.545+0000 7f7cd6800640 1 --2- 172.21.15.92:0/2328938581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 0x7f7cd80c8230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009d90 tx=0x7f7cd002e140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d13b558257ebe655 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.545+0000 7f7cd4a00640 1 -- 172.21.15.92:0/2328938581 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd003b070 con 0x7f7cd80c7da0 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.545+0000 7f7cd4a00640 1 -- 172.21.15.92:0/2328938581 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd0036440 con 0x7f7cd80c7da0 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.545+0000 7f7cd4a00640 1 -- 172.21.15.92:0/2328938581 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0036810 con 0x7f7cd80c7da0 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 -- 172.21.15.92:0/2328938581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 msgr2=0x7f7cd80c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 --2- 172.21.15.92:0/2328938581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 0x7f7cd80c8230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009d90 tx=0x7f7cd002e140 comp rx=0 tx=0).stop 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 -- 172.21.15.92:0/2328938581 shutdown_connections 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 --2- 172.21.15.92:0/2328938581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd80c7da0 0x7f7cd80c8230 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 --2- 172.21.15.92:0/2328938581 >> v2:172.21.15.92:3300/0 conn(0x7f7cd8151900 0x7f7cd8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 -- 172.21.15.92:0/2328938581 >> 172.21.15.92:0/2328938581 conn(0x7f7cd80c8d90 msgr2=0x7f7cd80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 -- 172.21.15.92:0/2328938581 shutdown_connections 2024-03-19T02:55:26.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.546+0000 7f7cdca00640 1 -- 172.21.15.92:0/2328938581 wait complete. 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.549+0000 7f7cdca00640 1 Processor -- start 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.549+0000 7f7cdca00640 1 -- start start 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 0x7f7cd81f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 0x7f7cd8156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 0x7f7cd814eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd80c9af0 con 0x7f7cd8151900 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd8071eb0 con 0x7f7cd80c7da0 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cdca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd80c9c70 con 0x7f7cd8151ce0 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cd5e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 0x7f7cd81f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cd6800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 0x7f7cd814eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cd5e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 0x7f7cd81f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:42204/0 (socket says 172.21.15.92:42204) 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cd5e00640 1 -- 172.21.15.92:0/3440553396 learned_addr learned my addr 172.21.15.92:0/3440553396 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.550+0000 7f7cd5400640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 0x7f7cd8156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 -- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 msgr2=0x7f7cd81f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 0x7f7cd81f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 -- 172.21.15.92:0/3440553396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 msgr2=0x7f7cd8156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 0x7f7cd8156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 -- 172.21.15.92:0/3440553396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd81f07d0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd5400640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 0x7f7cd8156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cd6800640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 0x7f7cd814eb80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8073840 tx=0x7f7cd0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd003b040 con 0x7f7cd8151ce0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd003f620 con 0x7f7cd8151ce0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd80cbab0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.551+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd003f9f0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.552+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd81f0a90 con 0x7f7cd8151ce0 2024-03-19T02:55:26.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.553+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd80684c0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.553+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7cd003fb90 con 0x7f7cd8151ce0 2024-03-19T02:55:26.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.554+0000 7f7cce000640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 0x7f7cb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.554+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7cd00ace50 con 0x7f7cd8151ce0 2024-03-19T02:55:26.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.554+0000 7f7cd5e00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 0x7f7cb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.555+0000 7f7cd5e00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 0x7f7cb406e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006fd0 tx=0x7f7cc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.558+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7cd007b8a0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.798 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.797+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cd80c4b80 con 0x7f7cd8151ce0 2024-03-19T02:55:26.799 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.798+0000 7f7cce000640 1 -- 172.21.15.92:0/3440553396 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f7cd007b1a0 con 0x7f7cd8151ce0 2024-03-19T02:55:26.799 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:26.800 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:55:26.800 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 msgr2=0x7f7cb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 0x7f7cb406e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006fd0 tx=0x7f7cc8008040 comp rx=0 tx=0).stop 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 msgr2=0x7f7cd814eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 0x7f7cd814eb80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8073840 tx=0x7f7cd0004a40 comp rx=0 tx=0).stop 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 shutdown_connections 2024-03-19T02:55:26.803 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7cb406bbd0 0x7f7cb406e090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8151ce0 0x7f7cd814eb80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cd8151900 0x7f7cd8156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 --2- 172.21.15.92:0/3440553396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd80c7da0 0x7f7cd81f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 >> 172.21.15.92:0/3440553396 conn(0x7f7cd80c8d90 msgr2=0x7f7cd814d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 shutdown_connections 2024-03-19T02:55:26.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:26.801+0000 7f7cdca00640 1 -- 172.21.15.92:0/3440553396 wait complete. 2024-03-19T02:55:26.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:55:27.021 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.020+0000 7f7040800640 1 Processor -- start 2024-03-19T02:55:27.021 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.020+0000 7f7040800640 1 -- start start 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 0x7f70380c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70381593d0 0x7f70380c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f70380c6c90 con 0x7f70381593d0 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7038159910 con 0x7f70380c6470 2024-03-19T02:55:27.023 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.021+0000 7f7040800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f703806e4a0 con 0x7f7038152bf0 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703d400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 0x7f70380c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:40224/0 (socket says 172.21.15.92:40224) 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 -- 172.21.15.92:0/1789130593 learned_addr learned my addr 172.21.15.92:0/1789130593 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703d400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 0x7f70380c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:42212/0 (socket says 172.21.15.92:42212) 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703e800640 1 --2- 172.21.15.92:0/1789130593 >> v2:172.21.15.92:3300/0 conn(0x7f70381593d0 0x7f70380c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 -- 172.21.15.92:0/1789130593 >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 msgr2=0x7f70380c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 --2- 172.21.15.92:0/1789130593 >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 0x7f70380c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 -- 172.21.15.92:0/1789130593 >> v2:172.21.15.92:3300/0 conn(0x7f70381593d0 msgr2=0x7f70380c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 --2- 172.21.15.92:0/1789130593 >> v2:172.21.15.92:3300/0 conn(0x7f70381593d0 0x7f70380c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.024 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.022+0000 7f703de00640 1 -- 172.21.15.92:0/1789130593 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70380c9a80 con 0x7f7038152bf0 2024-03-19T02:55:27.025 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.023+0000 7f703de00640 1 --2- 172.21.15.92:0/1789130593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7038075310 tx=0x7f702c02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=373703b99ccfbc0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.023+0000 7f703ca00640 1 -- 172.21.15.92:0/1789130593 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c02e070 con 0x7f7038152bf0 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.023+0000 7f703ca00640 1 -- 172.21.15.92:0/1789130593 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702c037070 con 0x7f7038152bf0 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.023+0000 7f703ca00640 1 -- 172.21.15.92:0/1789130593 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c031a00 con 0x7f7038152bf0 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 -- 172.21.15.92:0/1789130593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 msgr2=0x7f703806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 --2- 172.21.15.92:0/1789130593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7038075310 tx=0x7f702c02b880 comp rx=0 tx=0).stop 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 -- 172.21.15.92:0/1789130593 shutdown_connections 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 --2- 172.21.15.92:0/1789130593 >> v2:172.21.15.92:3300/0 conn(0x7f70381593d0 0x7f70380c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 --2- 172.21.15.92:0/1789130593 >> v2:172.21.15.159:3300/0 conn(0x7f70380c6470 0x7f70380c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 --2- 172.21.15.92:0/1789130593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7038152bf0 0x7f703806eef0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 -- 172.21.15.92:0/1789130593 >> 172.21.15.92:0/1789130593 conn(0x7f70380c8d90 msgr2=0x7f70380c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 -- 172.21.15.92:0/1789130593 shutdown_connections 2024-03-19T02:55:27.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.024+0000 7f7040800640 1 -- 172.21.15.92:0/1789130593 wait complete. 2024-03-19T02:55:27.027 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.026+0000 7f7040800640 1 Processor -- start 2024-03-19T02:55:27.027 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.026+0000 7f7040800640 1 -- start start 2024-03-19T02:55:27.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70380c6470 0x7f703815c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 0x7f70380cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70380c8930 con 0x7f70380c6470 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f703806ea60 con 0x7f7038152bf0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f7040800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f703806e6f0 con 0x7f70381593d0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70380c6470 0x7f703815c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 0x7f70380cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:42218/0 (socket says 172.21.15.92:42218) 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 -- 172.21.15.92:0/160059977 learned_addr learned my addr 172.21.15.92:0/160059977 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 -- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 msgr2=0x7f70380cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 0x7f70380cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 -- 172.21.15.92:0/160059977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70380c6470 msgr2=0x7f703815c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70380c6470 0x7f703815c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703d400640 1 -- 172.21.15.92:0/160059977 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703815cc70 con 0x7f7038152bf0 2024-03-19T02:55:27.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.027+0000 7f703e800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 0x7f70380cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f703d400640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7028012f00 tx=0x7f702800f480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702800d2d0 con 0x7f7038152bf0 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702801d070 con 0x7f7038152bf0 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70381f0240 con 0x7f7038152bf0 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702800fdd0 con 0x7f7038152bf0 2024-03-19T02:55:27.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.028+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70381f0800 con 0x7f7038152bf0 2024-03-19T02:55:27.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.029+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70381509e0 con 0x7f7038152bf0 2024-03-19T02:55:27.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.030+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7028017070 con 0x7f7038152bf0 2024-03-19T02:55:27.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.031+0000 7f7032000640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 0x7f701c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.031+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f702808e550 con 0x7f7038152bf0 2024-03-19T02:55:27.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.031+0000 7f703de00640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 0x7f701c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.033 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.032+0000 7f703de00640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 0x7f701c06e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f702c036000 tx=0x7f702c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.035+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f70381509e0 con 0x7f7038152bf0 2024-03-19T02:55:27.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.221+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f703814cc70 con 0x7f7038152bf0 2024-03-19T02:55:27.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.222+0000 7f7032000640 1 -- 172.21.15.92:0/160059977 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16605 (secure 0 0 0) 0x7f703814cc70 con 0x7f7038152bf0 2024-03-19T02:55:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:55:27.224 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T02:54:57.182173+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.224+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 msgr2=0x7f701c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.224+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 0x7f701c06e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f702c036000 tx=0x7f702c03d040 comp rx=0 tx=0).stop 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 msgr2=0x7f70381ecbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7028012f00 tx=0x7f702800f480 comp rx=0 tx=0).stop 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 shutdown_connections 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f701c06bbd0 0x7f701c06e090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70381593d0 0x7f70380cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038152bf0 0x7f70381ecbb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 --2- 172.21.15.92:0/160059977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70380c6470 0x7f703815c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 >> 172.21.15.92:0/160059977 conn(0x7f70380c8d90 msgr2=0x7f70381502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 shutdown_connections 2024-03-19T02:55:27.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:55:27.225+0000 7f7040800640 1 -- 172.21.15.92:0/160059977 wait complete. 2024-03-19T02:55:27.244 DEBUG:tasks.mds_thrash:Yielding 2024-03-19T02:55:27.245 INFO:teuthology.run_tasks:Running task workunit... 2024-03-19T02:55:27.259 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:55:27.260 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-19T02:55:27.260 INFO:tasks.workunit:timeout=6h 2024-03-19T02:55:27.260 INFO:tasks.workunit:cleanup=True 2024-03-19T02:55:27.260 DEBUG:teuthology.orchestra.run.smithi092:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:27.260 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-03-19T02:55:27.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2024-03-19T02:55:27.310 INFO:teuthology.orchestra.run.smithi092.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 2ah/42d Inode: 1 Links: 2 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-19T02:55:27.311 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-19 02:55:01.170396150 +0000 2024-03-19T02:55:27.312 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-19 02:55:17.357150618 +0000 2024-03-19T02:55:27.312 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-19 02:55:01.170396150 +0000 2024-03-19T02:55:27.312 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-19T02:55:27.312 DEBUG:teuthology.orchestra.run.smithi092:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-19T02:55:27.403 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:55:27.458 INFO:tasks.workunit.client.0.smithi092.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-19T02:55:28.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:55:28.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:55:58.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:55:58.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.378 INFO:tasks.workunit.client.0.smithi092.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: git switch -c 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr:Or undo this operation with: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: git switch - 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr: 2024-03-19T02:56:08.379 INFO:tasks.workunit.client.0.smithi092.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T02:56:08.389 DEBUG:teuthology.orchestra.run.smithi092:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-19T02:56:08.450 INFO:tasks.workunit.client.0.smithi092.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T02:56:08.458 INFO:tasks.workunit.client.0.smithi092.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:56:08.458 INFO:tasks.workunit.client.0.smithi092.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T02:56:09.297 INFO:tasks.workunit.client.0.smithi092.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T02:56:09.413 INFO:tasks.workunit.client.0.smithi092.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T02:56:09.498 INFO:tasks.workunit.client.0.smithi092.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:56:09.504 INFO:tasks.workunit.client.0.smithi092.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:56:09.504 INFO:tasks.workunit.client.0.smithi092.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T02:56:09.578 INFO:tasks.workunit.client.0.smithi092.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:56:09.586 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T02:56:09.586 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-19T02:56:09.631 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-03-19T02:56:09.633 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-03-19T02:56:09.633 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-19T02:56:09.746 INFO:tasks.workunit.client.0.smithi092.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-03-19T02:56:13.911 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir foo 2024-03-19T02:56:13.921 INFO:tasks.workunit.client.0.smithi092.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-03-19T02:56:14.022 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir foo/.snap/barsnap 2024-03-19T02:56:14.056 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-03-19T02:56:14.071 INFO:tasks.workunit.client.0.smithi092.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-03-19T02:56:15.121 INFO:tasks.workunit.client.0.smithi092.stdout:passed: files are identical 2024-03-19T02:56:15.122 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 'passed: files are identical' 2024-03-19T02:56:15.122 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir foo/.snap/barsnap 2024-03-19T02:56:15.170 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:56:15.170 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T02:56:15.171 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:56:15.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:56:15.288 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-03-19T02:56:15.288 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-19T02:56:15.344 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 1 2024-03-19T02:56:15.345 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 2 2024-03-19T02:56:15.346 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 3 2024-03-19T02:56:15.347 INFO:tasks.workunit.client.0.smithi092.stderr:+ '[' -e file4 ']' 2024-03-19T02:56:15.347 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir .snap/snap1 2024-03-19T02:56:15.380 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 4 2024-03-19T02:56:15.382 INFO:tasks.workunit.client.0.smithi092.stderr:++ ls 2024-03-19T02:56:15.388 INFO:tasks.workunit.client.0.smithi092.stderr:+ now='file1 2024-03-19T02:56:15.388 INFO:tasks.workunit.client.0.smithi092.stderr:file2 2024-03-19T02:56:15.388 INFO:tasks.workunit.client.0.smithi092.stderr:file3 2024-03-19T02:56:15.388 INFO:tasks.workunit.client.0.smithi092.stderr:file4' 2024-03-19T02:56:15.389 INFO:tasks.workunit.client.0.smithi092.stderr:++ ls .snap/snap1 2024-03-19T02:56:15.396 INFO:tasks.workunit.client.0.smithi092.stderr:+ then='file1 2024-03-19T02:56:15.396 INFO:tasks.workunit.client.0.smithi092.stderr:file2 2024-03-19T02:56:15.396 INFO:tasks.workunit.client.0.smithi092.stderr:file3' 2024-03-19T02:56:15.396 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir .snap/snap1 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:+ '[' 'file1 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:file2 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:file3 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:file4' = 'file1 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:file2 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:file3' ']' 2024-03-19T02:56:15.404 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 1 2024-03-19T02:56:15.411 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 2 2024-03-19T02:56:15.422 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 3 2024-03-19T02:56:15.437 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir .snap/snap1 2024-03-19T02:56:15.471 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 4 2024-03-19T02:56:15.480 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir .snap/snap1 2024-03-19T02:56:15.491 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm file1 file2 file3 file4 2024-03-19T02:56:15.501 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T02:56:15.501 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:56:15.503 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:56:15.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:56:15.596 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-03-19T02:56:15.597 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-19T02:56:15.667 INFO:tasks.workunit.client.0.smithi092.stdout:Create dir 100 to 199 ... 2024-03-19T02:56:15.681 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 100 2024-03-19T02:56:15.701 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 101 2024-03-19T02:56:15.721 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 102 2024-03-19T02:56:15.744 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 103 2024-03-19T02:56:15.767 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 104 2024-03-19T02:56:15.790 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 105 2024-03-19T02:56:15.813 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 106 2024-03-19T02:56:15.836 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 107 2024-03-19T02:56:15.857 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 108 2024-03-19T02:56:15.884 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 109 2024-03-19T02:56:15.911 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 110 2024-03-19T02:56:15.939 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 111 2024-03-19T02:56:15.968 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 112 2024-03-19T02:56:15.996 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 113 2024-03-19T02:56:16.024 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 114 2024-03-19T02:56:16.052 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 115 2024-03-19T02:56:16.080 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 116 2024-03-19T02:56:16.108 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 117 2024-03-19T02:56:16.135 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 118 2024-03-19T02:56:16.164 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 119 2024-03-19T02:56:16.191 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 120 2024-03-19T02:56:16.216 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 121 2024-03-19T02:56:16.242 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 122 2024-03-19T02:56:16.265 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 123 2024-03-19T02:56:16.288 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 124 2024-03-19T02:56:16.576 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 125 2024-03-19T02:56:16.860 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 126 2024-03-19T02:56:16.883 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 127 2024-03-19T02:56:16.906 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 128 2024-03-19T02:56:16.929 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 129 2024-03-19T02:56:16.953 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 130 2024-03-19T02:56:16.976 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 131 2024-03-19T02:56:17.002 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 132 2024-03-19T02:56:17.025 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 133 2024-03-19T02:56:17.049 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 134 2024-03-19T02:56:17.072 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 135 2024-03-19T02:56:17.097 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 136 2024-03-19T02:56:17.120 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 137 2024-03-19T02:56:17.144 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 138 2024-03-19T02:56:17.168 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 139 2024-03-19T02:56:17.191 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 140 2024-03-19T02:56:17.214 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 141 2024-03-19T02:56:17.239 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 142 2024-03-19T02:56:17.269 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 143 2024-03-19T02:56:17.299 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 144 2024-03-19T02:56:17.618 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 145 2024-03-19T02:56:17.642 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 146 2024-03-19T02:56:17.659 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 147 2024-03-19T02:56:17.676 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 148 2024-03-19T02:56:17.695 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 149 2024-03-19T02:56:17.713 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 150 2024-03-19T02:56:18.234 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 151 2024-03-19T02:56:18.258 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 152 2024-03-19T02:56:18.282 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 153 2024-03-19T02:56:18.305 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 154 2024-03-19T02:56:18.328 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 155 2024-03-19T02:56:18.352 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 156 2024-03-19T02:56:18.375 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 157 2024-03-19T02:56:18.399 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 158 2024-03-19T02:56:18.423 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 159 2024-03-19T02:56:18.447 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 160 2024-03-19T02:56:18.468 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 161 2024-03-19T02:56:18.492 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 162 2024-03-19T02:56:18.517 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 163 2024-03-19T02:56:18.540 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 164 2024-03-19T02:56:18.564 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 165 2024-03-19T02:56:19.474 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 166 2024-03-19T02:56:19.497 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 167 2024-03-19T02:56:19.520 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 168 2024-03-19T02:56:19.544 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 169 2024-03-19T02:56:20.326 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 170 2024-03-19T02:56:20.355 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 171 2024-03-19T02:56:20.380 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 172 2024-03-19T02:56:20.405 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 173 2024-03-19T02:56:20.429 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 174 2024-03-19T02:56:20.452 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 175 2024-03-19T02:56:20.476 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 176 2024-03-19T02:56:20.499 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 177 2024-03-19T02:56:20.521 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 178 2024-03-19T02:56:20.545 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 179 2024-03-19T02:56:20.569 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 180 2024-03-19T02:56:20.593 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 181 2024-03-19T02:56:20.617 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 182 2024-03-19T02:56:20.645 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 183 2024-03-19T02:56:20.669 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 184 2024-03-19T02:56:20.698 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 185 2024-03-19T02:56:20.729 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 186 2024-03-19T02:56:20.759 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 187 2024-03-19T02:56:20.790 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 188 2024-03-19T02:56:20.818 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 189 2024-03-19T02:56:20.843 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 190 2024-03-19T02:56:20.867 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 191 2024-03-19T02:56:20.890 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 192 2024-03-19T02:56:20.914 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 193 2024-03-19T02:56:20.939 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 194 2024-03-19T02:56:20.963 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 195 2024-03-19T02:56:20.989 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 196 2024-03-19T02:56:21.011 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 197 2024-03-19T02:56:21.035 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 198 2024-03-19T02:56:21.837 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 199 2024-03-19T02:56:21.868 INFO:tasks.workunit.client.0.smithi092.stdout:Take first snapshot .snap/test1 2024-03-19T02:56:23.742 INFO:tasks.workunit.client.0.smithi092.stdout:Create dir 200 to 299 ... 2024-03-19T02:56:23.746 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 200 2024-03-19T02:56:23.768 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 201 2024-03-19T02:56:23.790 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 202 2024-03-19T02:56:23.812 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 203 2024-03-19T02:56:23.835 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 204 2024-03-19T02:56:23.856 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 205 2024-03-19T02:56:23.878 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 206 2024-03-19T02:56:23.903 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 207 2024-03-19T02:56:23.928 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 208 2024-03-19T02:56:23.949 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 209 2024-03-19T02:56:24.232 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 210 2024-03-19T02:56:24.254 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 211 2024-03-19T02:56:24.276 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 212 2024-03-19T02:56:24.299 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 213 2024-03-19T02:56:24.322 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 214 2024-03-19T02:56:24.344 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 215 2024-03-19T02:56:24.366 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 216 2024-03-19T02:56:24.388 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 217 2024-03-19T02:56:24.414 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 218 2024-03-19T02:56:24.441 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 219 2024-03-19T02:56:24.467 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 220 2024-03-19T02:56:24.495 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 221 2024-03-19T02:56:24.522 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 222 2024-03-19T02:56:24.548 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 223 2024-03-19T02:56:24.573 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 224 2024-03-19T02:56:24.603 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 225 2024-03-19T02:56:24.626 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 226 2024-03-19T02:56:24.647 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 227 2024-03-19T02:56:24.673 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 228 2024-03-19T02:56:24.697 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 229 2024-03-19T02:56:24.720 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 230 2024-03-19T02:56:24.743 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 231 2024-03-19T02:56:24.767 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 232 2024-03-19T02:56:24.789 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 233 2024-03-19T02:56:24.815 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 234 2024-03-19T02:56:24.842 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 235 2024-03-19T02:56:24.869 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 236 2024-03-19T02:56:24.894 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 237 2024-03-19T02:56:24.916 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 238 2024-03-19T02:56:24.940 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 239 2024-03-19T02:56:24.962 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 240 2024-03-19T02:56:24.989 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 241 2024-03-19T02:56:25.015 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 242 2024-03-19T02:56:25.304 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 243 2024-03-19T02:56:25.327 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 244 2024-03-19T02:56:25.348 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 245 2024-03-19T02:56:25.369 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 246 2024-03-19T02:56:25.389 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 247 2024-03-19T02:56:25.611 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 248 2024-03-19T02:56:25.633 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 249 2024-03-19T02:56:25.921 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 250 2024-03-19T02:56:25.950 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 251 2024-03-19T02:56:27.283 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 252 2024-03-19T02:56:27.308 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 253 2024-03-19T02:56:27.332 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 254 2024-03-19T02:56:27.355 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 255 2024-03-19T02:56:27.377 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 256 2024-03-19T02:56:27.399 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 257 2024-03-19T02:56:27.424 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 258 2024-03-19T02:56:27.452 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 259 2024-03-19T02:56:27.479 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 260 2024-03-19T02:56:27.507 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 261 2024-03-19T02:56:27.530 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 262 2024-03-19T02:56:27.555 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 263 2024-03-19T02:56:27.582 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 264 2024-03-19T02:56:27.605 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 265 2024-03-19T02:56:27.625 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 266 2024-03-19T02:56:27.647 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 267 2024-03-19T02:56:27.669 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 268 2024-03-19T02:56:27.692 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 269 2024-03-19T02:56:27.714 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 270 2024-03-19T02:56:27.736 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 271 2024-03-19T02:56:27.758 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 272 2024-03-19T02:56:27.780 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 273 2024-03-19T02:56:27.803 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 274 2024-03-19T02:56:27.825 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 275 2024-03-19T02:56:27.847 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 276 2024-03-19T02:56:27.871 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 277 2024-03-19T02:56:27.894 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 278 2024-03-19T02:56:27.918 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 279 2024-03-19T02:56:27.941 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 280 2024-03-19T02:56:27.963 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 281 2024-03-19T02:56:27.991 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 282 2024-03-19T02:56:28.016 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 283 2024-03-19T02:56:28.039 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 284 2024-03-19T02:56:28.063 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 285 2024-03-19T02:56:28.085 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 286 2024-03-19T02:56:28.109 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 287 2024-03-19T02:56:28.132 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 288 2024-03-19T02:56:28.424 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 289 2024-03-19T02:56:28.448 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 290 2024-03-19T02:56:28.732 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 291 2024-03-19T02:56:28.758 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 292 2024-03-19T02:56:28.785 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 293 2024-03-19T02:56:28.810 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 294 2024-03-19T02:56:28.837 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 295 2024-03-19T02:56:28.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:56:28.863 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 296 2024-03-19T02:56:28.870 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:56:29.128 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 297 2024-03-19T02:56:29.151 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 298 2024-03-19T02:56:29.173 INFO:tasks.workunit.client.0.smithi092.stdout: create dir 299 2024-03-19T02:56:29.195 INFO:tasks.workunit.client.0.smithi092.stdout:Create a snapshot in every first level dir ... 2024-03-19T02:56:29.373 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/.snap/snap-subdir-test 2024-03-19T02:56:29.422 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/30 file after the snapshot 2024-03-19T02:56:29.426 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/31 file after the snapshot 2024-03-19T02:56:29.429 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/32 file after the snapshot 2024-03-19T02:56:29.430 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/33 file after the snapshot 2024-03-19T02:56:29.431 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/34 file after the snapshot 2024-03-19T02:56:29.433 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/35 file after the snapshot 2024-03-19T02:56:29.434 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/36 file after the snapshot 2024-03-19T02:56:29.435 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/37 file after the snapshot 2024-03-19T02:56:29.437 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/38 file after the snapshot 2024-03-19T02:56:29.438 INFO:tasks.workunit.client.0.smithi092.stdout: create 100/39 file after the snapshot 2024-03-19T02:56:29.439 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/.snap/snap-subdir-test 2024-03-19T02:56:29.493 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/30 file after the snapshot 2024-03-19T02:56:29.498 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/31 file after the snapshot 2024-03-19T02:56:29.500 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/32 file after the snapshot 2024-03-19T02:56:29.501 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/33 file after the snapshot 2024-03-19T02:56:29.502 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/34 file after the snapshot 2024-03-19T02:56:29.504 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/35 file after the snapshot 2024-03-19T02:56:29.505 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/36 file after the snapshot 2024-03-19T02:56:29.506 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/37 file after the snapshot 2024-03-19T02:56:29.508 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/38 file after the snapshot 2024-03-19T02:56:29.510 INFO:tasks.workunit.client.0.smithi092.stdout: create 101/39 file after the snapshot 2024-03-19T02:56:29.512 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/.snap/snap-subdir-test 2024-03-19T02:56:29.967 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/30 file after the snapshot 2024-03-19T02:56:29.980 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/31 file after the snapshot 2024-03-19T02:56:29.982 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/32 file after the snapshot 2024-03-19T02:56:29.986 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/33 file after the snapshot 2024-03-19T02:56:29.986 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/34 file after the snapshot 2024-03-19T02:56:29.988 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/35 file after the snapshot 2024-03-19T02:56:29.990 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/36 file after the snapshot 2024-03-19T02:56:29.992 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/37 file after the snapshot 2024-03-19T02:56:29.994 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/38 file after the snapshot 2024-03-19T02:56:29.996 INFO:tasks.workunit.client.0.smithi092.stdout: create 102/39 file after the snapshot 2024-03-19T02:56:29.998 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/.snap/snap-subdir-test 2024-03-19T02:56:30.070 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/30 file after the snapshot 2024-03-19T02:56:30.074 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/31 file after the snapshot 2024-03-19T02:56:30.076 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/32 file after the snapshot 2024-03-19T02:56:30.077 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/33 file after the snapshot 2024-03-19T02:56:30.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/34 file after the snapshot 2024-03-19T02:56:30.527 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/35 file after the snapshot 2024-03-19T02:56:30.529 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/36 file after the snapshot 2024-03-19T02:56:30.531 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/37 file after the snapshot 2024-03-19T02:56:30.533 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/38 file after the snapshot 2024-03-19T02:56:30.535 INFO:tasks.workunit.client.0.smithi092.stdout: create 103/39 file after the snapshot 2024-03-19T02:56:30.536 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/.snap/snap-subdir-test 2024-03-19T02:56:30.594 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/30 file after the snapshot 2024-03-19T02:56:30.606 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/31 file after the snapshot 2024-03-19T02:56:30.607 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/32 file after the snapshot 2024-03-19T02:56:30.609 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/33 file after the snapshot 2024-03-19T02:56:30.610 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/34 file after the snapshot 2024-03-19T02:56:30.612 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/35 file after the snapshot 2024-03-19T02:56:30.613 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/36 file after the snapshot 2024-03-19T02:56:30.615 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/37 file after the snapshot 2024-03-19T02:56:30.616 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/38 file after the snapshot 2024-03-19T02:56:30.618 INFO:tasks.workunit.client.0.smithi092.stdout: create 104/39 file after the snapshot 2024-03-19T02:56:30.619 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/.snap/snap-subdir-test 2024-03-19T02:56:30.678 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/30 file after the snapshot 2024-03-19T02:56:30.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/31 file after the snapshot 2024-03-19T02:56:30.685 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/32 file after the snapshot 2024-03-19T02:56:30.686 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/33 file after the snapshot 2024-03-19T02:56:30.687 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/34 file after the snapshot 2024-03-19T02:56:30.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/35 file after the snapshot 2024-03-19T02:56:30.690 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/36 file after the snapshot 2024-03-19T02:56:30.692 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/37 file after the snapshot 2024-03-19T02:56:30.694 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/38 file after the snapshot 2024-03-19T02:56:30.695 INFO:tasks.workunit.client.0.smithi092.stdout: create 105/39 file after the snapshot 2024-03-19T02:56:30.697 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/.snap/snap-subdir-test 2024-03-19T02:56:30.767 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/30 file after the snapshot 2024-03-19T02:56:30.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/31 file after the snapshot 2024-03-19T02:56:30.775 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/32 file after the snapshot 2024-03-19T02:56:30.776 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/33 file after the snapshot 2024-03-19T02:56:30.778 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/34 file after the snapshot 2024-03-19T02:56:30.779 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/35 file after the snapshot 2024-03-19T02:56:30.781 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/36 file after the snapshot 2024-03-19T02:56:30.782 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/37 file after the snapshot 2024-03-19T02:56:30.784 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/38 file after the snapshot 2024-03-19T02:56:30.785 INFO:tasks.workunit.client.0.smithi092.stdout: create 106/39 file after the snapshot 2024-03-19T02:56:30.787 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/.snap/snap-subdir-test 2024-03-19T02:56:30.852 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/30 file after the snapshot 2024-03-19T02:56:30.866 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/31 file after the snapshot 2024-03-19T02:56:30.868 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/32 file after the snapshot 2024-03-19T02:56:30.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/33 file after the snapshot 2024-03-19T02:56:30.872 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/34 file after the snapshot 2024-03-19T02:56:30.874 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/35 file after the snapshot 2024-03-19T02:56:30.875 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/36 file after the snapshot 2024-03-19T02:56:30.877 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/37 file after the snapshot 2024-03-19T02:56:30.878 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/38 file after the snapshot 2024-03-19T02:56:30.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 107/39 file after the snapshot 2024-03-19T02:56:30.881 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/.snap/snap-subdir-test 2024-03-19T02:56:30.950 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/30 file after the snapshot 2024-03-19T02:56:30.954 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/31 file after the snapshot 2024-03-19T02:56:30.956 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/32 file after the snapshot 2024-03-19T02:56:30.957 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/33 file after the snapshot 2024-03-19T02:56:30.959 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/34 file after the snapshot 2024-03-19T02:56:30.960 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/35 file after the snapshot 2024-03-19T02:56:30.961 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/36 file after the snapshot 2024-03-19T02:56:30.963 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/37 file after the snapshot 2024-03-19T02:56:30.964 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/38 file after the snapshot 2024-03-19T02:56:30.966 INFO:tasks.workunit.client.0.smithi092.stdout: create 108/39 file after the snapshot 2024-03-19T02:56:30.967 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/.snap/snap-subdir-test 2024-03-19T02:56:31.022 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/30 file after the snapshot 2024-03-19T02:56:31.028 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/31 file after the snapshot 2024-03-19T02:56:31.029 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/32 file after the snapshot 2024-03-19T02:56:31.031 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/33 file after the snapshot 2024-03-19T02:56:31.032 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/34 file after the snapshot 2024-03-19T02:56:31.034 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/35 file after the snapshot 2024-03-19T02:56:31.035 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/36 file after the snapshot 2024-03-19T02:56:31.036 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/37 file after the snapshot 2024-03-19T02:56:31.039 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/38 file after the snapshot 2024-03-19T02:56:31.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 109/39 file after the snapshot 2024-03-19T02:56:31.041 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/.snap/snap-subdir-test 2024-03-19T02:56:31.086 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/30 file after the snapshot 2024-03-19T02:56:31.099 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/31 file after the snapshot 2024-03-19T02:56:31.101 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/32 file after the snapshot 2024-03-19T02:56:31.103 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/33 file after the snapshot 2024-03-19T02:56:31.105 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/34 file after the snapshot 2024-03-19T02:56:31.107 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/35 file after the snapshot 2024-03-19T02:56:31.109 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/36 file after the snapshot 2024-03-19T02:56:31.111 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/37 file after the snapshot 2024-03-19T02:56:31.113 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/38 file after the snapshot 2024-03-19T02:56:31.115 INFO:tasks.workunit.client.0.smithi092.stdout: create 110/39 file after the snapshot 2024-03-19T02:56:31.116 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/.snap/snap-subdir-test 2024-03-19T02:56:31.163 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/30 file after the snapshot 2024-03-19T02:56:31.425 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/31 file after the snapshot 2024-03-19T02:56:31.427 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/32 file after the snapshot 2024-03-19T02:56:31.428 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/33 file after the snapshot 2024-03-19T02:56:31.430 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/34 file after the snapshot 2024-03-19T02:56:31.431 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/35 file after the snapshot 2024-03-19T02:56:31.433 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/36 file after the snapshot 2024-03-19T02:56:31.434 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/37 file after the snapshot 2024-03-19T02:56:31.436 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/38 file after the snapshot 2024-03-19T02:56:31.438 INFO:tasks.workunit.client.0.smithi092.stdout: create 111/39 file after the snapshot 2024-03-19T02:56:31.440 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/.snap/snap-subdir-test 2024-03-19T02:56:31.498 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/30 file after the snapshot 2024-03-19T02:56:31.511 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/31 file after the snapshot 2024-03-19T02:56:31.512 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/32 file after the snapshot 2024-03-19T02:56:31.514 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/33 file after the snapshot 2024-03-19T02:56:31.516 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/34 file after the snapshot 2024-03-19T02:56:31.518 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/35 file after the snapshot 2024-03-19T02:56:31.520 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/36 file after the snapshot 2024-03-19T02:56:31.521 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/37 file after the snapshot 2024-03-19T02:56:31.523 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/38 file after the snapshot 2024-03-19T02:56:31.525 INFO:tasks.workunit.client.0.smithi092.stdout: create 112/39 file after the snapshot 2024-03-19T02:56:31.527 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/.snap/snap-subdir-test 2024-03-19T02:56:31.570 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/30 file after the snapshot 2024-03-19T02:56:31.581 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/31 file after the snapshot 2024-03-19T02:56:31.587 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/32 file after the snapshot 2024-03-19T02:56:31.589 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/33 file after the snapshot 2024-03-19T02:56:31.591 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/34 file after the snapshot 2024-03-19T02:56:31.593 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/35 file after the snapshot 2024-03-19T02:56:31.595 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/36 file after the snapshot 2024-03-19T02:56:31.597 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/37 file after the snapshot 2024-03-19T02:56:31.599 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/38 file after the snapshot 2024-03-19T02:56:31.601 INFO:tasks.workunit.client.0.smithi092.stdout: create 113/39 file after the snapshot 2024-03-19T02:56:31.602 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/.snap/snap-subdir-test 2024-03-19T02:56:31.665 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/30 file after the snapshot 2024-03-19T02:56:31.670 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/31 file after the snapshot 2024-03-19T02:56:31.671 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/32 file after the snapshot 2024-03-19T02:56:31.673 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/33 file after the snapshot 2024-03-19T02:56:31.674 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/34 file after the snapshot 2024-03-19T02:56:31.676 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/35 file after the snapshot 2024-03-19T02:56:31.677 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/36 file after the snapshot 2024-03-19T02:56:31.678 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/37 file after the snapshot 2024-03-19T02:56:31.680 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/38 file after the snapshot 2024-03-19T02:56:31.681 INFO:tasks.workunit.client.0.smithi092.stdout: create 114/39 file after the snapshot 2024-03-19T02:56:31.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/.snap/snap-subdir-test 2024-03-19T02:56:31.730 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/30 file after the snapshot 2024-03-19T02:56:31.806 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/31 file after the snapshot 2024-03-19T02:56:31.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/32 file after the snapshot 2024-03-19T02:56:32.057 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/33 file after the snapshot 2024-03-19T02:56:32.058 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/34 file after the snapshot 2024-03-19T02:56:32.059 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/35 file after the snapshot 2024-03-19T02:56:32.061 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/36 file after the snapshot 2024-03-19T02:56:32.062 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/37 file after the snapshot 2024-03-19T02:56:32.064 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/38 file after the snapshot 2024-03-19T02:56:32.065 INFO:tasks.workunit.client.0.smithi092.stdout: create 115/39 file after the snapshot 2024-03-19T02:56:32.067 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/.snap/snap-subdir-test 2024-03-19T02:56:32.115 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/30 file after the snapshot 2024-03-19T02:56:32.119 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/31 file after the snapshot 2024-03-19T02:56:32.121 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/32 file after the snapshot 2024-03-19T02:56:32.123 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/33 file after the snapshot 2024-03-19T02:56:32.126 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/34 file after the snapshot 2024-03-19T02:56:32.129 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/35 file after the snapshot 2024-03-19T02:56:32.131 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/36 file after the snapshot 2024-03-19T02:56:32.133 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/37 file after the snapshot 2024-03-19T02:56:32.135 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/38 file after the snapshot 2024-03-19T02:56:32.137 INFO:tasks.workunit.client.0.smithi092.stdout: create 116/39 file after the snapshot 2024-03-19T02:56:32.139 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/.snap/snap-subdir-test 2024-03-19T02:56:32.897 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/30 file after the snapshot 2024-03-19T02:56:33.478 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/31 file after the snapshot 2024-03-19T02:56:33.479 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/32 file after the snapshot 2024-03-19T02:56:33.481 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/33 file after the snapshot 2024-03-19T02:56:33.482 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/34 file after the snapshot 2024-03-19T02:56:33.484 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/35 file after the snapshot 2024-03-19T02:56:33.485 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/36 file after the snapshot 2024-03-19T02:56:33.486 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/37 file after the snapshot 2024-03-19T02:56:33.488 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/38 file after the snapshot 2024-03-19T02:56:33.489 INFO:tasks.workunit.client.0.smithi092.stdout: create 117/39 file after the snapshot 2024-03-19T02:56:33.491 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/.snap/snap-subdir-test 2024-03-19T02:56:33.535 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/30 file after the snapshot 2024-03-19T02:56:33.548 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/31 file after the snapshot 2024-03-19T02:56:33.550 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/32 file after the snapshot 2024-03-19T02:56:33.552 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/33 file after the snapshot 2024-03-19T02:56:33.554 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/34 file after the snapshot 2024-03-19T02:56:33.556 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/35 file after the snapshot 2024-03-19T02:56:33.558 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/36 file after the snapshot 2024-03-19T02:56:33.560 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/37 file after the snapshot 2024-03-19T02:56:33.562 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/38 file after the snapshot 2024-03-19T02:56:33.564 INFO:tasks.workunit.client.0.smithi092.stdout: create 118/39 file after the snapshot 2024-03-19T02:56:33.565 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/.snap/snap-subdir-test 2024-03-19T02:56:33.618 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/30 file after the snapshot 2024-03-19T02:56:33.622 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/31 file after the snapshot 2024-03-19T02:56:33.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/32 file after the snapshot 2024-03-19T02:56:33.624 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/33 file after the snapshot 2024-03-19T02:56:33.626 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/34 file after the snapshot 2024-03-19T02:56:33.627 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/35 file after the snapshot 2024-03-19T02:56:33.628 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/36 file after the snapshot 2024-03-19T02:56:33.630 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/37 file after the snapshot 2024-03-19T02:56:33.631 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/38 file after the snapshot 2024-03-19T02:56:33.632 INFO:tasks.workunit.client.0.smithi092.stdout: create 119/39 file after the snapshot 2024-03-19T02:56:33.634 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/.snap/snap-subdir-test 2024-03-19T02:56:33.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/30 file after the snapshot 2024-03-19T02:56:33.879 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/31 file after the snapshot 2024-03-19T02:56:33.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/32 file after the snapshot 2024-03-19T02:56:33.881 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/33 file after the snapshot 2024-03-19T02:56:33.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/34 file after the snapshot 2024-03-19T02:56:33.884 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/35 file after the snapshot 2024-03-19T02:56:33.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/36 file after the snapshot 2024-03-19T02:56:33.886 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/37 file after the snapshot 2024-03-19T02:56:33.888 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/38 file after the snapshot 2024-03-19T02:56:33.889 INFO:tasks.workunit.client.0.smithi092.stdout: create 120/39 file after the snapshot 2024-03-19T02:56:33.891 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/.snap/snap-subdir-test 2024-03-19T02:56:33.929 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/30 file after the snapshot 2024-03-19T02:56:33.944 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/31 file after the snapshot 2024-03-19T02:56:33.945 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/32 file after the snapshot 2024-03-19T02:56:33.947 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/33 file after the snapshot 2024-03-19T02:56:33.949 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/34 file after the snapshot 2024-03-19T02:56:33.951 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/35 file after the snapshot 2024-03-19T02:56:33.953 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/36 file after the snapshot 2024-03-19T02:56:33.955 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/37 file after the snapshot 2024-03-19T02:56:33.957 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/38 file after the snapshot 2024-03-19T02:56:33.959 INFO:tasks.workunit.client.0.smithi092.stdout: create 121/39 file after the snapshot 2024-03-19T02:56:33.961 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/.snap/snap-subdir-test 2024-03-19T02:56:34.002 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/30 file after the snapshot 2024-03-19T02:56:34.008 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/31 file after the snapshot 2024-03-19T02:56:34.010 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/32 file after the snapshot 2024-03-19T02:56:34.011 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/33 file after the snapshot 2024-03-19T02:56:34.013 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/34 file after the snapshot 2024-03-19T02:56:34.014 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/35 file after the snapshot 2024-03-19T02:56:34.015 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/36 file after the snapshot 2024-03-19T02:56:34.017 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/37 file after the snapshot 2024-03-19T02:56:34.018 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/38 file after the snapshot 2024-03-19T02:56:34.019 INFO:tasks.workunit.client.0.smithi092.stdout: create 122/39 file after the snapshot 2024-03-19T02:56:34.021 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/.snap/snap-subdir-test 2024-03-19T02:56:34.063 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/30 file after the snapshot 2024-03-19T02:56:34.068 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/31 file after the snapshot 2024-03-19T02:56:34.070 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/32 file after the snapshot 2024-03-19T02:56:34.071 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/33 file after the snapshot 2024-03-19T02:56:34.073 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/34 file after the snapshot 2024-03-19T02:56:34.074 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/35 file after the snapshot 2024-03-19T02:56:34.075 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/36 file after the snapshot 2024-03-19T02:56:34.076 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/37 file after the snapshot 2024-03-19T02:56:34.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/38 file after the snapshot 2024-03-19T02:56:34.079 INFO:tasks.workunit.client.0.smithi092.stdout: create 123/39 file after the snapshot 2024-03-19T02:56:34.080 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/.snap/snap-subdir-test 2024-03-19T02:56:34.126 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/30 file after the snapshot 2024-03-19T02:56:34.130 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/31 file after the snapshot 2024-03-19T02:56:34.131 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/32 file after the snapshot 2024-03-19T02:56:34.132 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/33 file after the snapshot 2024-03-19T02:56:34.134 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/34 file after the snapshot 2024-03-19T02:56:34.135 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/35 file after the snapshot 2024-03-19T02:56:34.137 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/36 file after the snapshot 2024-03-19T02:56:34.138 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/37 file after the snapshot 2024-03-19T02:56:34.140 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/38 file after the snapshot 2024-03-19T02:56:34.141 INFO:tasks.workunit.client.0.smithi092.stdout: create 124/39 file after the snapshot 2024-03-19T02:56:34.142 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/.snap/snap-subdir-test 2024-03-19T02:56:34.185 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/30 file after the snapshot 2024-03-19T02:56:34.458 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/31 file after the snapshot 2024-03-19T02:56:34.460 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/32 file after the snapshot 2024-03-19T02:56:34.462 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/33 file after the snapshot 2024-03-19T02:56:34.465 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/34 file after the snapshot 2024-03-19T02:56:34.467 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/35 file after the snapshot 2024-03-19T02:56:34.469 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/36 file after the snapshot 2024-03-19T02:56:34.471 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/37 file after the snapshot 2024-03-19T02:56:34.473 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/38 file after the snapshot 2024-03-19T02:56:34.474 INFO:tasks.workunit.client.0.smithi092.stdout: create 125/39 file after the snapshot 2024-03-19T02:56:34.476 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/.snap/snap-subdir-test 2024-03-19T02:56:34.543 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/30 file after the snapshot 2024-03-19T02:56:34.549 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/31 file after the snapshot 2024-03-19T02:56:34.550 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/32 file after the snapshot 2024-03-19T02:56:34.552 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/33 file after the snapshot 2024-03-19T02:56:34.553 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/34 file after the snapshot 2024-03-19T02:56:34.555 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/35 file after the snapshot 2024-03-19T02:56:34.556 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/36 file after the snapshot 2024-03-19T02:56:34.558 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/37 file after the snapshot 2024-03-19T02:56:34.559 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/38 file after the snapshot 2024-03-19T02:56:34.561 INFO:tasks.workunit.client.0.smithi092.stdout: create 126/39 file after the snapshot 2024-03-19T02:56:34.562 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/.snap/snap-subdir-test 2024-03-19T02:56:34.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/30 file after the snapshot 2024-03-19T02:56:34.679 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/31 file after the snapshot 2024-03-19T02:56:34.680 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/32 file after the snapshot 2024-03-19T02:56:34.682 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/33 file after the snapshot 2024-03-19T02:56:34.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/34 file after the snapshot 2024-03-19T02:56:34.685 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/35 file after the snapshot 2024-03-19T02:56:34.686 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/36 file after the snapshot 2024-03-19T02:56:34.688 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/37 file after the snapshot 2024-03-19T02:56:34.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/38 file after the snapshot 2024-03-19T02:56:34.690 INFO:tasks.workunit.client.0.smithi092.stdout: create 127/39 file after the snapshot 2024-03-19T02:56:34.692 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/.snap/snap-subdir-test 2024-03-19T02:56:35.547 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/30 file after the snapshot 2024-03-19T02:56:35.888 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/31 file after the snapshot 2024-03-19T02:56:36.074 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/32 file after the snapshot 2024-03-19T02:56:36.075 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/33 file after the snapshot 2024-03-19T02:56:36.077 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/34 file after the snapshot 2024-03-19T02:56:36.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/35 file after the snapshot 2024-03-19T02:56:36.080 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/36 file after the snapshot 2024-03-19T02:56:36.081 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/37 file after the snapshot 2024-03-19T02:56:36.083 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/38 file after the snapshot 2024-03-19T02:56:36.084 INFO:tasks.workunit.client.0.smithi092.stdout: create 128/39 file after the snapshot 2024-03-19T02:56:36.085 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/.snap/snap-subdir-test 2024-03-19T02:56:36.137 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/30 file after the snapshot 2024-03-19T02:56:36.142 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/31 file after the snapshot 2024-03-19T02:56:36.143 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/32 file after the snapshot 2024-03-19T02:56:36.145 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/33 file after the snapshot 2024-03-19T02:56:36.146 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/34 file after the snapshot 2024-03-19T02:56:36.148 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/35 file after the snapshot 2024-03-19T02:56:36.149 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/36 file after the snapshot 2024-03-19T02:56:36.151 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/37 file after the snapshot 2024-03-19T02:56:36.153 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/38 file after the snapshot 2024-03-19T02:56:36.154 INFO:tasks.workunit.client.0.smithi092.stdout: create 129/39 file after the snapshot 2024-03-19T02:56:36.156 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/.snap/snap-subdir-test 2024-03-19T02:56:36.206 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/30 file after the snapshot 2024-03-19T02:56:36.211 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/31 file after the snapshot 2024-03-19T02:56:36.212 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/32 file after the snapshot 2024-03-19T02:56:36.214 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/33 file after the snapshot 2024-03-19T02:56:36.215 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/34 file after the snapshot 2024-03-19T02:56:36.217 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/35 file after the snapshot 2024-03-19T02:56:36.218 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/36 file after the snapshot 2024-03-19T02:56:36.220 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/37 file after the snapshot 2024-03-19T02:56:36.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/38 file after the snapshot 2024-03-19T02:56:36.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 130/39 file after the snapshot 2024-03-19T02:56:36.224 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/.snap/snap-subdir-test 2024-03-19T02:56:36.273 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/30 file after the snapshot 2024-03-19T02:56:36.279 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/31 file after the snapshot 2024-03-19T02:56:36.280 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/32 file after the snapshot 2024-03-19T02:56:36.281 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/33 file after the snapshot 2024-03-19T02:56:36.283 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/34 file after the snapshot 2024-03-19T02:56:36.284 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/35 file after the snapshot 2024-03-19T02:56:36.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/36 file after the snapshot 2024-03-19T02:56:36.287 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/37 file after the snapshot 2024-03-19T02:56:36.289 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/38 file after the snapshot 2024-03-19T02:56:36.290 INFO:tasks.workunit.client.0.smithi092.stdout: create 131/39 file after the snapshot 2024-03-19T02:56:36.291 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/.snap/snap-subdir-test 2024-03-19T02:56:36.352 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/30 file after the snapshot 2024-03-19T02:56:36.367 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/31 file after the snapshot 2024-03-19T02:56:36.369 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/32 file after the snapshot 2024-03-19T02:56:36.371 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/33 file after the snapshot 2024-03-19T02:56:36.373 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/34 file after the snapshot 2024-03-19T02:56:36.375 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/35 file after the snapshot 2024-03-19T02:56:36.377 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/36 file after the snapshot 2024-03-19T02:56:36.379 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/37 file after the snapshot 2024-03-19T02:56:36.381 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/38 file after the snapshot 2024-03-19T02:56:36.383 INFO:tasks.workunit.client.0.smithi092.stdout: create 132/39 file after the snapshot 2024-03-19T02:56:36.385 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/.snap/snap-subdir-test 2024-03-19T02:56:36.447 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/30 file after the snapshot 2024-03-19T02:56:36.452 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/31 file after the snapshot 2024-03-19T02:56:36.454 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/32 file after the snapshot 2024-03-19T02:56:36.455 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/33 file after the snapshot 2024-03-19T02:56:36.456 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/34 file after the snapshot 2024-03-19T02:56:36.458 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/35 file after the snapshot 2024-03-19T02:56:36.459 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/36 file after the snapshot 2024-03-19T02:56:36.460 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/37 file after the snapshot 2024-03-19T02:56:36.462 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/38 file after the snapshot 2024-03-19T02:56:36.463 INFO:tasks.workunit.client.0.smithi092.stdout: create 133/39 file after the snapshot 2024-03-19T02:56:36.464 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/.snap/snap-subdir-test 2024-03-19T02:56:36.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/30 file after the snapshot 2024-03-19T02:56:36.688 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/31 file after the snapshot 2024-03-19T02:56:36.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/32 file after the snapshot 2024-03-19T02:56:36.691 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/33 file after the snapshot 2024-03-19T02:56:36.692 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/34 file after the snapshot 2024-03-19T02:56:36.693 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/35 file after the snapshot 2024-03-19T02:56:36.695 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/36 file after the snapshot 2024-03-19T02:56:36.696 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/37 file after the snapshot 2024-03-19T02:56:36.698 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/38 file after the snapshot 2024-03-19T02:56:36.699 INFO:tasks.workunit.client.0.smithi092.stdout: create 134/39 file after the snapshot 2024-03-19T02:56:36.701 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/.snap/snap-subdir-test 2024-03-19T02:56:36.763 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/30 file after the snapshot 2024-03-19T02:56:36.767 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/31 file after the snapshot 2024-03-19T02:56:36.769 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/32 file after the snapshot 2024-03-19T02:56:36.770 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/33 file after the snapshot 2024-03-19T02:56:36.771 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/34 file after the snapshot 2024-03-19T02:56:36.773 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/35 file after the snapshot 2024-03-19T02:56:36.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/36 file after the snapshot 2024-03-19T02:56:36.775 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/37 file after the snapshot 2024-03-19T02:56:36.777 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/38 file after the snapshot 2024-03-19T02:56:36.778 INFO:tasks.workunit.client.0.smithi092.stdout: create 135/39 file after the snapshot 2024-03-19T02:56:36.780 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/.snap/snap-subdir-test 2024-03-19T02:56:36.824 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/30 file after the snapshot 2024-03-19T02:56:36.830 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/31 file after the snapshot 2024-03-19T02:56:36.832 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/32 file after the snapshot 2024-03-19T02:56:36.833 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/33 file after the snapshot 2024-03-19T02:56:36.835 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/34 file after the snapshot 2024-03-19T02:56:36.836 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/35 file after the snapshot 2024-03-19T02:56:36.838 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/36 file after the snapshot 2024-03-19T02:56:36.839 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/37 file after the snapshot 2024-03-19T02:56:36.841 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/38 file after the snapshot 2024-03-19T02:56:36.842 INFO:tasks.workunit.client.0.smithi092.stdout: create 136/39 file after the snapshot 2024-03-19T02:56:36.844 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/.snap/snap-subdir-test 2024-03-19T02:56:36.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/30 file after the snapshot 2024-03-19T02:56:36.892 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/31 file after the snapshot 2024-03-19T02:56:36.894 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/32 file after the snapshot 2024-03-19T02:56:36.895 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/33 file after the snapshot 2024-03-19T02:56:36.897 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/34 file after the snapshot 2024-03-19T02:56:36.898 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/35 file after the snapshot 2024-03-19T02:56:36.899 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/36 file after the snapshot 2024-03-19T02:56:36.901 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/37 file after the snapshot 2024-03-19T02:56:36.903 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/38 file after the snapshot 2024-03-19T02:56:36.904 INFO:tasks.workunit.client.0.smithi092.stdout: create 137/39 file after the snapshot 2024-03-19T02:56:36.906 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/.snap/snap-subdir-test 2024-03-19T02:56:36.946 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/30 file after the snapshot 2024-03-19T02:56:36.949 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/31 file after the snapshot 2024-03-19T02:56:36.955 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/32 file after the snapshot 2024-03-19T02:56:36.956 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/33 file after the snapshot 2024-03-19T02:56:36.958 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/34 file after the snapshot 2024-03-19T02:56:36.959 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/35 file after the snapshot 2024-03-19T02:56:37.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/36 file after the snapshot 2024-03-19T02:56:37.691 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/37 file after the snapshot 2024-03-19T02:56:37.693 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/38 file after the snapshot 2024-03-19T02:56:37.695 INFO:tasks.workunit.client.0.smithi092.stdout: create 138/39 file after the snapshot 2024-03-19T02:56:37.696 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/.snap/snap-subdir-test 2024-03-19T02:56:37.738 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/30 file after the snapshot 2024-03-19T02:56:37.744 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/31 file after the snapshot 2024-03-19T02:56:37.746 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/32 file after the snapshot 2024-03-19T02:56:37.747 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/33 file after the snapshot 2024-03-19T02:56:37.748 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/34 file after the snapshot 2024-03-19T02:56:37.750 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/35 file after the snapshot 2024-03-19T02:56:37.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/36 file after the snapshot 2024-03-19T02:56:37.753 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/37 file after the snapshot 2024-03-19T02:56:37.754 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/38 file after the snapshot 2024-03-19T02:56:37.756 INFO:tasks.workunit.client.0.smithi092.stdout: create 139/39 file after the snapshot 2024-03-19T02:56:37.757 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/.snap/snap-subdir-test 2024-03-19T02:56:37.808 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/30 file after the snapshot 2024-03-19T02:56:37.871 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/31 file after the snapshot 2024-03-19T02:56:38.743 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/32 file after the snapshot 2024-03-19T02:56:38.744 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/33 file after the snapshot 2024-03-19T02:56:38.746 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/34 file after the snapshot 2024-03-19T02:56:38.747 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/35 file after the snapshot 2024-03-19T02:56:38.748 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/36 file after the snapshot 2024-03-19T02:56:38.750 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/37 file after the snapshot 2024-03-19T02:56:38.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/38 file after the snapshot 2024-03-19T02:56:38.753 INFO:tasks.workunit.client.0.smithi092.stdout: create 140/39 file after the snapshot 2024-03-19T02:56:38.754 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/.snap/snap-subdir-test 2024-03-19T02:56:38.795 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/30 file after the snapshot 2024-03-19T02:56:38.800 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/31 file after the snapshot 2024-03-19T02:56:38.802 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/32 file after the snapshot 2024-03-19T02:56:38.803 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/33 file after the snapshot 2024-03-19T02:56:38.804 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/34 file after the snapshot 2024-03-19T02:56:38.806 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/35 file after the snapshot 2024-03-19T02:56:38.807 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/36 file after the snapshot 2024-03-19T02:56:38.809 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/37 file after the snapshot 2024-03-19T02:56:38.810 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/38 file after the snapshot 2024-03-19T02:56:38.812 INFO:tasks.workunit.client.0.smithi092.stdout: create 141/39 file after the snapshot 2024-03-19T02:56:38.813 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/.snap/snap-subdir-test 2024-03-19T02:56:38.871 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/30 file after the snapshot 2024-03-19T02:56:38.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/31 file after the snapshot 2024-03-19T02:56:38.878 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/32 file after the snapshot 2024-03-19T02:56:38.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/33 file after the snapshot 2024-03-19T02:56:38.882 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/34 file after the snapshot 2024-03-19T02:56:38.884 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/35 file after the snapshot 2024-03-19T02:56:38.886 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/36 file after the snapshot 2024-03-19T02:56:38.888 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/37 file after the snapshot 2024-03-19T02:56:38.889 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/38 file after the snapshot 2024-03-19T02:56:38.891 INFO:tasks.workunit.client.0.smithi092.stdout: create 142/39 file after the snapshot 2024-03-19T02:56:38.893 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/.snap/snap-subdir-test 2024-03-19T02:56:38.936 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/30 file after the snapshot 2024-03-19T02:56:39.646 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/31 file after the snapshot 2024-03-19T02:56:39.647 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/32 file after the snapshot 2024-03-19T02:56:39.649 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/33 file after the snapshot 2024-03-19T02:56:39.650 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/34 file after the snapshot 2024-03-19T02:56:39.652 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/35 file after the snapshot 2024-03-19T02:56:39.653 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/36 file after the snapshot 2024-03-19T02:56:39.654 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/37 file after the snapshot 2024-03-19T02:56:39.656 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/38 file after the snapshot 2024-03-19T02:56:39.658 INFO:tasks.workunit.client.0.smithi092.stdout: create 143/39 file after the snapshot 2024-03-19T02:56:39.659 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/.snap/snap-subdir-test 2024-03-19T02:56:39.719 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/30 file after the snapshot 2024-03-19T02:56:39.725 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/31 file after the snapshot 2024-03-19T02:56:39.727 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/32 file after the snapshot 2024-03-19T02:56:39.728 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/33 file after the snapshot 2024-03-19T02:56:39.730 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/34 file after the snapshot 2024-03-19T02:56:39.731 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/35 file after the snapshot 2024-03-19T02:56:39.732 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/36 file after the snapshot 2024-03-19T02:56:39.734 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/37 file after the snapshot 2024-03-19T02:56:39.735 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/38 file after the snapshot 2024-03-19T02:56:39.736 INFO:tasks.workunit.client.0.smithi092.stdout: create 144/39 file after the snapshot 2024-03-19T02:56:39.738 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/.snap/snap-subdir-test 2024-03-19T02:56:39.792 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/30 file after the snapshot 2024-03-19T02:56:39.797 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/31 file after the snapshot 2024-03-19T02:56:39.798 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/32 file after the snapshot 2024-03-19T02:56:39.799 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/33 file after the snapshot 2024-03-19T02:56:39.801 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/34 file after the snapshot 2024-03-19T02:56:39.802 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/35 file after the snapshot 2024-03-19T02:56:39.804 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/36 file after the snapshot 2024-03-19T02:56:39.806 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/37 file after the snapshot 2024-03-19T02:56:39.807 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/38 file after the snapshot 2024-03-19T02:56:39.808 INFO:tasks.workunit.client.0.smithi092.stdout: create 145/39 file after the snapshot 2024-03-19T02:56:39.810 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/.snap/snap-subdir-test 2024-03-19T02:56:39.865 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/30 file after the snapshot 2024-03-19T02:56:39.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/31 file after the snapshot 2024-03-19T02:56:40.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/32 file after the snapshot 2024-03-19T02:56:40.081 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/33 file after the snapshot 2024-03-19T02:56:40.083 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/34 file after the snapshot 2024-03-19T02:56:40.084 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/35 file after the snapshot 2024-03-19T02:56:40.086 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/36 file after the snapshot 2024-03-19T02:56:40.087 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/37 file after the snapshot 2024-03-19T02:56:40.089 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/38 file after the snapshot 2024-03-19T02:56:40.090 INFO:tasks.workunit.client.0.smithi092.stdout: create 146/39 file after the snapshot 2024-03-19T02:56:40.240 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/.snap/snap-subdir-test 2024-03-19T02:56:40.295 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/30 file after the snapshot 2024-03-19T02:56:40.554 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/31 file after the snapshot 2024-03-19T02:56:40.555 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/32 file after the snapshot 2024-03-19T02:56:40.557 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/33 file after the snapshot 2024-03-19T02:56:40.558 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/34 file after the snapshot 2024-03-19T02:56:40.560 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/35 file after the snapshot 2024-03-19T02:56:40.561 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/36 file after the snapshot 2024-03-19T02:56:40.563 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/37 file after the snapshot 2024-03-19T02:56:40.565 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/38 file after the snapshot 2024-03-19T02:56:40.566 INFO:tasks.workunit.client.0.smithi092.stdout: create 147/39 file after the snapshot 2024-03-19T02:56:40.568 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/.snap/snap-subdir-test 2024-03-19T02:56:40.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/30 file after the snapshot 2024-03-19T02:56:40.626 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/31 file after the snapshot 2024-03-19T02:56:40.628 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/32 file after the snapshot 2024-03-19T02:56:40.629 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/33 file after the snapshot 2024-03-19T02:56:40.631 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/34 file after the snapshot 2024-03-19T02:56:40.632 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/35 file after the snapshot 2024-03-19T02:56:40.633 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/36 file after the snapshot 2024-03-19T02:56:40.635 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/37 file after the snapshot 2024-03-19T02:56:40.636 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/38 file after the snapshot 2024-03-19T02:56:40.637 INFO:tasks.workunit.client.0.smithi092.stdout: create 148/39 file after the snapshot 2024-03-19T02:56:40.641 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/.snap/snap-subdir-test 2024-03-19T02:56:40.694 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/30 file after the snapshot 2024-03-19T02:56:40.700 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/31 file after the snapshot 2024-03-19T02:56:40.701 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/32 file after the snapshot 2024-03-19T02:56:40.702 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/33 file after the snapshot 2024-03-19T02:56:40.704 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/34 file after the snapshot 2024-03-19T02:56:40.705 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/35 file after the snapshot 2024-03-19T02:56:40.706 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/36 file after the snapshot 2024-03-19T02:56:40.708 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/37 file after the snapshot 2024-03-19T02:56:40.709 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/38 file after the snapshot 2024-03-19T02:56:40.710 INFO:tasks.workunit.client.0.smithi092.stdout: create 149/39 file after the snapshot 2024-03-19T02:56:40.712 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/.snap/snap-subdir-test 2024-03-19T02:56:40.766 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/30 file after the snapshot 2024-03-19T02:56:40.779 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/31 file after the snapshot 2024-03-19T02:56:40.781 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/32 file after the snapshot 2024-03-19T02:56:40.782 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/33 file after the snapshot 2024-03-19T02:56:40.784 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/34 file after the snapshot 2024-03-19T02:56:40.786 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/35 file after the snapshot 2024-03-19T02:56:40.787 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/36 file after the snapshot 2024-03-19T02:56:40.789 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/37 file after the snapshot 2024-03-19T02:56:40.790 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/38 file after the snapshot 2024-03-19T02:56:40.791 INFO:tasks.workunit.client.0.smithi092.stdout: create 150/39 file after the snapshot 2024-03-19T02:56:40.792 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/.snap/snap-subdir-test 2024-03-19T02:56:40.867 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/30 file after the snapshot 2024-03-19T02:56:40.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/31 file after the snapshot 2024-03-19T02:56:40.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/32 file after the snapshot 2024-03-19T02:56:40.871 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/33 file after the snapshot 2024-03-19T02:56:40.872 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/34 file after the snapshot 2024-03-19T02:56:40.873 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/35 file after the snapshot 2024-03-19T02:56:40.874 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/36 file after the snapshot 2024-03-19T02:56:40.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/37 file after the snapshot 2024-03-19T02:56:40.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/38 file after the snapshot 2024-03-19T02:56:40.898 INFO:tasks.workunit.client.0.smithi092.stdout: create 151/39 file after the snapshot 2024-03-19T02:56:40.904 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/.snap/snap-subdir-test 2024-03-19T02:56:41.026 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/30 file after the snapshot 2024-03-19T02:56:41.033 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/31 file after the snapshot 2024-03-19T02:56:41.035 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/32 file after the snapshot 2024-03-19T02:56:41.036 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/33 file after the snapshot 2024-03-19T02:56:41.037 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/34 file after the snapshot 2024-03-19T02:56:41.038 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/35 file after the snapshot 2024-03-19T02:56:41.039 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/36 file after the snapshot 2024-03-19T02:56:41.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/37 file after the snapshot 2024-03-19T02:56:41.041 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/38 file after the snapshot 2024-03-19T02:56:41.043 INFO:tasks.workunit.client.0.smithi092.stdout: create 152/39 file after the snapshot 2024-03-19T02:56:41.044 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/.snap/snap-subdir-test 2024-03-19T02:56:41.105 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/30 file after the snapshot 2024-03-19T02:56:41.111 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/31 file after the snapshot 2024-03-19T02:56:41.112 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/32 file after the snapshot 2024-03-19T02:56:41.112 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/33 file after the snapshot 2024-03-19T02:56:41.113 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/34 file after the snapshot 2024-03-19T02:56:41.115 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/35 file after the snapshot 2024-03-19T02:56:41.115 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/36 file after the snapshot 2024-03-19T02:56:41.116 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/37 file after the snapshot 2024-03-19T02:56:41.118 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/38 file after the snapshot 2024-03-19T02:56:41.119 INFO:tasks.workunit.client.0.smithi092.stdout: create 153/39 file after the snapshot 2024-03-19T02:56:41.120 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/.snap/snap-subdir-test 2024-03-19T02:56:41.156 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/30 file after the snapshot 2024-03-19T02:56:41.166 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/31 file after the snapshot 2024-03-19T02:56:41.167 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/32 file after the snapshot 2024-03-19T02:56:41.168 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/33 file after the snapshot 2024-03-19T02:56:41.170 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/34 file after the snapshot 2024-03-19T02:56:41.171 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/35 file after the snapshot 2024-03-19T02:56:41.172 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/36 file after the snapshot 2024-03-19T02:56:41.174 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/37 file after the snapshot 2024-03-19T02:56:41.175 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/38 file after the snapshot 2024-03-19T02:56:41.176 INFO:tasks.workunit.client.0.smithi092.stdout: create 154/39 file after the snapshot 2024-03-19T02:56:41.178 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/.snap/snap-subdir-test 2024-03-19T02:56:41.219 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/30 file after the snapshot 2024-03-19T02:56:41.229 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/31 file after the snapshot 2024-03-19T02:56:41.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/32 file after the snapshot 2024-03-19T02:56:41.231 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/33 file after the snapshot 2024-03-19T02:56:41.232 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/34 file after the snapshot 2024-03-19T02:56:41.233 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/35 file after the snapshot 2024-03-19T02:56:41.234 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/36 file after the snapshot 2024-03-19T02:56:41.235 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/37 file after the snapshot 2024-03-19T02:56:41.236 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/38 file after the snapshot 2024-03-19T02:56:41.238 INFO:tasks.workunit.client.0.smithi092.stdout: create 155/39 file after the snapshot 2024-03-19T02:56:41.239 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/.snap/snap-subdir-test 2024-03-19T02:56:41.278 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/30 file after the snapshot 2024-03-19T02:56:41.312 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/31 file after the snapshot 2024-03-19T02:56:41.316 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/32 file after the snapshot 2024-03-19T02:56:41.333 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/33 file after the snapshot 2024-03-19T02:56:41.350 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/34 file after the snapshot 2024-03-19T02:56:41.662 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/35 file after the snapshot 2024-03-19T02:56:41.665 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/36 file after the snapshot 2024-03-19T02:56:41.665 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/37 file after the snapshot 2024-03-19T02:56:41.666 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/38 file after the snapshot 2024-03-19T02:56:41.667 INFO:tasks.workunit.client.0.smithi092.stdout: create 156/39 file after the snapshot 2024-03-19T02:56:41.669 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/.snap/snap-subdir-test 2024-03-19T02:56:41.726 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/30 file after the snapshot 2024-03-19T02:56:41.728 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/31 file after the snapshot 2024-03-19T02:56:41.731 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/32 file after the snapshot 2024-03-19T02:56:41.732 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/33 file after the snapshot 2024-03-19T02:56:41.734 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/34 file after the snapshot 2024-03-19T02:56:41.735 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/35 file after the snapshot 2024-03-19T02:56:41.737 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/36 file after the snapshot 2024-03-19T02:56:41.738 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/37 file after the snapshot 2024-03-19T02:56:41.740 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/38 file after the snapshot 2024-03-19T02:56:41.741 INFO:tasks.workunit.client.0.smithi092.stdout: create 157/39 file after the snapshot 2024-03-19T02:56:41.743 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/.snap/snap-subdir-test 2024-03-19T02:56:41.786 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/30 file after the snapshot 2024-03-19T02:56:42.053 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/31 file after the snapshot 2024-03-19T02:56:42.058 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/32 file after the snapshot 2024-03-19T02:56:42.060 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/33 file after the snapshot 2024-03-19T02:56:42.062 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/34 file after the snapshot 2024-03-19T02:56:42.064 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/35 file after the snapshot 2024-03-19T02:56:42.065 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/36 file after the snapshot 2024-03-19T02:56:42.067 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/37 file after the snapshot 2024-03-19T02:56:42.069 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/38 file after the snapshot 2024-03-19T02:56:42.071 INFO:tasks.workunit.client.0.smithi092.stdout: create 158/39 file after the snapshot 2024-03-19T02:56:42.073 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/.snap/snap-subdir-test 2024-03-19T02:56:42.116 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/30 file after the snapshot 2024-03-19T02:56:42.121 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/31 file after the snapshot 2024-03-19T02:56:42.122 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/32 file after the snapshot 2024-03-19T02:56:42.124 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/33 file after the snapshot 2024-03-19T02:56:42.125 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/34 file after the snapshot 2024-03-19T02:56:42.127 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/35 file after the snapshot 2024-03-19T02:56:42.128 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/36 file after the snapshot 2024-03-19T02:56:42.130 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/37 file after the snapshot 2024-03-19T02:56:42.131 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/38 file after the snapshot 2024-03-19T02:56:42.132 INFO:tasks.workunit.client.0.smithi092.stdout: create 159/39 file after the snapshot 2024-03-19T02:56:42.134 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/.snap/snap-subdir-test 2024-03-19T02:56:42.177 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/30 file after the snapshot 2024-03-19T02:56:42.863 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/31 file after the snapshot 2024-03-19T02:56:43.104 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/32 file after the snapshot 2024-03-19T02:56:43.105 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/33 file after the snapshot 2024-03-19T02:56:43.106 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/34 file after the snapshot 2024-03-19T02:56:43.108 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/35 file after the snapshot 2024-03-19T02:56:43.109 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/36 file after the snapshot 2024-03-19T02:56:43.111 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/37 file after the snapshot 2024-03-19T02:56:43.112 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/38 file after the snapshot 2024-03-19T02:56:43.113 INFO:tasks.workunit.client.0.smithi092.stdout: create 160/39 file after the snapshot 2024-03-19T02:56:43.115 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/.snap/snap-subdir-test 2024-03-19T02:56:43.161 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/30 file after the snapshot 2024-03-19T02:56:43.166 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/31 file after the snapshot 2024-03-19T02:56:43.167 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/32 file after the snapshot 2024-03-19T02:56:43.168 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/33 file after the snapshot 2024-03-19T02:56:43.170 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/34 file after the snapshot 2024-03-19T02:56:43.171 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/35 file after the snapshot 2024-03-19T02:56:43.172 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/36 file after the snapshot 2024-03-19T02:56:43.174 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/37 file after the snapshot 2024-03-19T02:56:43.175 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/38 file after the snapshot 2024-03-19T02:56:43.176 INFO:tasks.workunit.client.0.smithi092.stdout: create 161/39 file after the snapshot 2024-03-19T02:56:43.177 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/.snap/snap-subdir-test 2024-03-19T02:56:43.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/30 file after the snapshot 2024-03-19T02:56:43.481 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/31 file after the snapshot 2024-03-19T02:56:43.482 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/32 file after the snapshot 2024-03-19T02:56:43.484 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/33 file after the snapshot 2024-03-19T02:56:43.485 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/34 file after the snapshot 2024-03-19T02:56:43.487 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/35 file after the snapshot 2024-03-19T02:56:43.488 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/36 file after the snapshot 2024-03-19T02:56:43.489 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/37 file after the snapshot 2024-03-19T02:56:43.491 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/38 file after the snapshot 2024-03-19T02:56:43.492 INFO:tasks.workunit.client.0.smithi092.stdout: create 162/39 file after the snapshot 2024-03-19T02:56:43.494 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/.snap/snap-subdir-test 2024-03-19T02:56:43.541 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/30 file after the snapshot 2024-03-19T02:56:43.553 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/31 file after the snapshot 2024-03-19T02:56:43.555 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/32 file after the snapshot 2024-03-19T02:56:43.557 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/33 file after the snapshot 2024-03-19T02:56:43.559 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/34 file after the snapshot 2024-03-19T02:56:43.561 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/35 file after the snapshot 2024-03-19T02:56:43.563 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/36 file after the snapshot 2024-03-19T02:56:43.565 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/37 file after the snapshot 2024-03-19T02:56:43.567 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/38 file after the snapshot 2024-03-19T02:56:43.569 INFO:tasks.workunit.client.0.smithi092.stdout: create 163/39 file after the snapshot 2024-03-19T02:56:43.571 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/.snap/snap-subdir-test 2024-03-19T02:56:43.617 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/30 file after the snapshot 2024-03-19T02:56:43.882 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/31 file after the snapshot 2024-03-19T02:56:43.888 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/32 file after the snapshot 2024-03-19T02:56:44.386 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/33 file after the snapshot 2024-03-19T02:56:44.387 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/34 file after the snapshot 2024-03-19T02:56:44.389 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/35 file after the snapshot 2024-03-19T02:56:44.390 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/36 file after the snapshot 2024-03-19T02:56:44.392 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/37 file after the snapshot 2024-03-19T02:56:44.393 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/38 file after the snapshot 2024-03-19T02:56:44.395 INFO:tasks.workunit.client.0.smithi092.stdout: create 164/39 file after the snapshot 2024-03-19T02:56:44.396 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/.snap/snap-subdir-test 2024-03-19T02:56:44.441 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/30 file after the snapshot 2024-03-19T02:56:44.446 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/31 file after the snapshot 2024-03-19T02:56:44.448 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/32 file after the snapshot 2024-03-19T02:56:44.449 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/33 file after the snapshot 2024-03-19T02:56:44.454 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/34 file after the snapshot 2024-03-19T02:56:44.454 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/35 file after the snapshot 2024-03-19T02:56:44.454 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/36 file after the snapshot 2024-03-19T02:56:44.515 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/37 file after the snapshot 2024-03-19T02:56:44.516 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/38 file after the snapshot 2024-03-19T02:56:44.517 INFO:tasks.workunit.client.0.smithi092.stdout: create 165/39 file after the snapshot 2024-03-19T02:56:44.519 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/.snap/snap-subdir-test 2024-03-19T02:56:44.849 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/30 file after the snapshot 2024-03-19T02:56:44.856 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/31 file after the snapshot 2024-03-19T02:56:44.857 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/32 file after the snapshot 2024-03-19T02:56:44.859 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/33 file after the snapshot 2024-03-19T02:56:44.860 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/34 file after the snapshot 2024-03-19T02:56:44.861 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/35 file after the snapshot 2024-03-19T02:56:44.863 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/36 file after the snapshot 2024-03-19T02:56:44.864 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/37 file after the snapshot 2024-03-19T02:56:44.866 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/38 file after the snapshot 2024-03-19T02:56:44.867 INFO:tasks.workunit.client.0.smithi092.stdout: create 166/39 file after the snapshot 2024-03-19T02:56:44.869 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/.snap/snap-subdir-test 2024-03-19T02:56:44.917 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/30 file after the snapshot 2024-03-19T02:56:44.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/31 file after the snapshot 2024-03-19T02:56:44.924 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/32 file after the snapshot 2024-03-19T02:56:44.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/33 file after the snapshot 2024-03-19T02:56:44.927 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/34 file after the snapshot 2024-03-19T02:56:44.929 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/35 file after the snapshot 2024-03-19T02:56:44.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/36 file after the snapshot 2024-03-19T02:56:44.932 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/37 file after the snapshot 2024-03-19T02:56:44.934 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/38 file after the snapshot 2024-03-19T02:56:44.935 INFO:tasks.workunit.client.0.smithi092.stdout: create 167/39 file after the snapshot 2024-03-19T02:56:44.936 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/.snap/snap-subdir-test 2024-03-19T02:56:44.986 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/30 file after the snapshot 2024-03-19T02:56:44.992 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/31 file after the snapshot 2024-03-19T02:56:44.993 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/32 file after the snapshot 2024-03-19T02:56:44.994 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/33 file after the snapshot 2024-03-19T02:56:44.996 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/34 file after the snapshot 2024-03-19T02:56:44.997 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/35 file after the snapshot 2024-03-19T02:56:44.999 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/36 file after the snapshot 2024-03-19T02:56:45.000 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/37 file after the snapshot 2024-03-19T02:56:45.001 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/38 file after the snapshot 2024-03-19T02:56:45.003 INFO:tasks.workunit.client.0.smithi092.stdout: create 168/39 file after the snapshot 2024-03-19T02:56:45.004 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/.snap/snap-subdir-test 2024-03-19T02:56:45.056 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/30 file after the snapshot 2024-03-19T02:56:45.068 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/31 file after the snapshot 2024-03-19T02:56:45.070 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/32 file after the snapshot 2024-03-19T02:56:45.072 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/33 file after the snapshot 2024-03-19T02:56:45.074 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/34 file after the snapshot 2024-03-19T02:56:45.076 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/35 file after the snapshot 2024-03-19T02:56:45.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/36 file after the snapshot 2024-03-19T02:56:45.080 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/37 file after the snapshot 2024-03-19T02:56:45.082 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/38 file after the snapshot 2024-03-19T02:56:45.084 INFO:tasks.workunit.client.0.smithi092.stdout: create 169/39 file after the snapshot 2024-03-19T02:56:45.086 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/.snap/snap-subdir-test 2024-03-19T02:56:45.137 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/30 file after the snapshot 2024-03-19T02:56:45.141 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/31 file after the snapshot 2024-03-19T02:56:45.143 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/32 file after the snapshot 2024-03-19T02:56:45.144 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/33 file after the snapshot 2024-03-19T02:56:45.145 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/34 file after the snapshot 2024-03-19T02:56:45.147 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/35 file after the snapshot 2024-03-19T02:56:45.148 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/36 file after the snapshot 2024-03-19T02:56:45.150 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/37 file after the snapshot 2024-03-19T02:56:45.151 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/38 file after the snapshot 2024-03-19T02:56:45.153 INFO:tasks.workunit.client.0.smithi092.stdout: create 170/39 file after the snapshot 2024-03-19T02:56:45.154 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/.snap/snap-subdir-test 2024-03-19T02:56:45.216 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/30 file after the snapshot 2024-03-19T02:56:45.219 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/31 file after the snapshot 2024-03-19T02:56:45.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/32 file after the snapshot 2024-03-19T02:56:45.222 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/33 file after the snapshot 2024-03-19T02:56:45.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/34 file after the snapshot 2024-03-19T02:56:45.225 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/35 file after the snapshot 2024-03-19T02:56:45.226 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/36 file after the snapshot 2024-03-19T02:56:45.228 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/37 file after the snapshot 2024-03-19T02:56:45.229 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/38 file after the snapshot 2024-03-19T02:56:45.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 171/39 file after the snapshot 2024-03-19T02:56:45.232 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/.snap/snap-subdir-test 2024-03-19T02:56:45.531 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/30 file after the snapshot 2024-03-19T02:56:45.697 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/31 file after the snapshot 2024-03-19T02:56:45.699 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/32 file after the snapshot 2024-03-19T02:56:45.701 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/33 file after the snapshot 2024-03-19T02:56:45.703 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/34 file after the snapshot 2024-03-19T02:56:45.704 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/35 file after the snapshot 2024-03-19T02:56:45.706 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/36 file after the snapshot 2024-03-19T02:56:45.708 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/37 file after the snapshot 2024-03-19T02:56:45.710 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/38 file after the snapshot 2024-03-19T02:56:45.712 INFO:tasks.workunit.client.0.smithi092.stdout: create 172/39 file after the snapshot 2024-03-19T02:56:45.714 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/.snap/snap-subdir-test 2024-03-19T02:56:45.766 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/30 file after the snapshot 2024-03-19T02:56:46.055 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/31 file after the snapshot 2024-03-19T02:56:46.068 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/32 file after the snapshot 2024-03-19T02:56:46.216 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/33 file after the snapshot 2024-03-19T02:56:46.217 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/34 file after the snapshot 2024-03-19T02:56:46.218 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/35 file after the snapshot 2024-03-19T02:56:46.220 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/36 file after the snapshot 2024-03-19T02:56:46.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/37 file after the snapshot 2024-03-19T02:56:46.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/38 file after the snapshot 2024-03-19T02:56:46.224 INFO:tasks.workunit.client.0.smithi092.stdout: create 173/39 file after the snapshot 2024-03-19T02:56:46.225 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/.snap/snap-subdir-test 2024-03-19T02:56:46.290 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/30 file after the snapshot 2024-03-19T02:56:46.294 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/31 file after the snapshot 2024-03-19T02:56:46.296 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/32 file after the snapshot 2024-03-19T02:56:46.297 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/33 file after the snapshot 2024-03-19T02:56:46.299 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/34 file after the snapshot 2024-03-19T02:56:46.300 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/35 file after the snapshot 2024-03-19T02:56:46.301 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/36 file after the snapshot 2024-03-19T02:56:46.303 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/37 file after the snapshot 2024-03-19T02:56:46.304 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/38 file after the snapshot 2024-03-19T02:56:46.306 INFO:tasks.workunit.client.0.smithi092.stdout: create 174/39 file after the snapshot 2024-03-19T02:56:46.307 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/.snap/snap-subdir-test 2024-03-19T02:56:46.368 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/30 file after the snapshot 2024-03-19T02:56:46.374 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/31 file after the snapshot 2024-03-19T02:56:46.375 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/32 file after the snapshot 2024-03-19T02:56:46.376 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/33 file after the snapshot 2024-03-19T02:56:46.378 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/34 file after the snapshot 2024-03-19T02:56:46.381 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/35 file after the snapshot 2024-03-19T02:56:46.382 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/36 file after the snapshot 2024-03-19T02:56:46.382 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/37 file after the snapshot 2024-03-19T02:56:46.383 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/38 file after the snapshot 2024-03-19T02:56:46.385 INFO:tasks.workunit.client.0.smithi092.stdout: create 175/39 file after the snapshot 2024-03-19T02:56:46.387 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/.snap/snap-subdir-test 2024-03-19T02:56:46.911 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/30 file after the snapshot 2024-03-19T02:56:47.170 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/31 file after the snapshot 2024-03-19T02:56:47.172 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/32 file after the snapshot 2024-03-19T02:56:47.433 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/33 file after the snapshot 2024-03-19T02:56:47.435 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/34 file after the snapshot 2024-03-19T02:56:47.436 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/35 file after the snapshot 2024-03-19T02:56:47.438 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/36 file after the snapshot 2024-03-19T02:56:47.439 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/37 file after the snapshot 2024-03-19T02:56:47.441 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/38 file after the snapshot 2024-03-19T02:56:47.442 INFO:tasks.workunit.client.0.smithi092.stdout: create 176/39 file after the snapshot 2024-03-19T02:56:47.443 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/.snap/snap-subdir-test 2024-03-19T02:56:47.476 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/30 file after the snapshot 2024-03-19T02:56:47.490 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/31 file after the snapshot 2024-03-19T02:56:47.491 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/32 file after the snapshot 2024-03-19T02:56:47.494 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/33 file after the snapshot 2024-03-19T02:56:47.495 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/34 file after the snapshot 2024-03-19T02:56:47.497 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/35 file after the snapshot 2024-03-19T02:56:47.499 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/36 file after the snapshot 2024-03-19T02:56:47.501 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/37 file after the snapshot 2024-03-19T02:56:47.503 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/38 file after the snapshot 2024-03-19T02:56:47.505 INFO:tasks.workunit.client.0.smithi092.stdout: create 177/39 file after the snapshot 2024-03-19T02:56:47.507 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/.snap/snap-subdir-test 2024-03-19T02:56:47.545 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/30 file after the snapshot 2024-03-19T02:56:47.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/31 file after the snapshot 2024-03-19T02:56:48.333 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/32 file after the snapshot 2024-03-19T02:56:48.335 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/33 file after the snapshot 2024-03-19T02:56:48.337 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/34 file after the snapshot 2024-03-19T02:56:48.339 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/35 file after the snapshot 2024-03-19T02:56:48.341 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/36 file after the snapshot 2024-03-19T02:56:48.343 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/37 file after the snapshot 2024-03-19T02:56:48.345 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/38 file after the snapshot 2024-03-19T02:56:48.347 INFO:tasks.workunit.client.0.smithi092.stdout: create 178/39 file after the snapshot 2024-03-19T02:56:48.349 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/.snap/snap-subdir-test 2024-03-19T02:56:48.894 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/30 file after the snapshot 2024-03-19T02:56:48.907 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/31 file after the snapshot 2024-03-19T02:56:48.908 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/32 file after the snapshot 2024-03-19T02:56:49.176 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/33 file after the snapshot 2024-03-19T02:56:49.177 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/34 file after the snapshot 2024-03-19T02:56:49.179 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/35 file after the snapshot 2024-03-19T02:56:49.180 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/36 file after the snapshot 2024-03-19T02:56:49.181 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/37 file after the snapshot 2024-03-19T02:56:49.183 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/38 file after the snapshot 2024-03-19T02:56:49.184 INFO:tasks.workunit.client.0.smithi092.stdout: create 179/39 file after the snapshot 2024-03-19T02:56:49.186 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/.snap/snap-subdir-test 2024-03-19T02:56:49.225 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/30 file after the snapshot 2024-03-19T02:56:49.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/31 file after the snapshot 2024-03-19T02:56:49.231 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/32 file after the snapshot 2024-03-19T02:56:49.232 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/33 file after the snapshot 2024-03-19T02:56:49.234 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/34 file after the snapshot 2024-03-19T02:56:49.235 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/35 file after the snapshot 2024-03-19T02:56:49.237 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/36 file after the snapshot 2024-03-19T02:56:49.238 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/37 file after the snapshot 2024-03-19T02:56:49.240 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/38 file after the snapshot 2024-03-19T02:56:49.241 INFO:tasks.workunit.client.0.smithi092.stdout: create 180/39 file after the snapshot 2024-03-19T02:56:49.243 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/.snap/snap-subdir-test 2024-03-19T02:56:49.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/30 file after the snapshot 2024-03-19T02:56:49.299 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/31 file after the snapshot 2024-03-19T02:56:49.301 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/32 file after the snapshot 2024-03-19T02:56:49.302 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/33 file after the snapshot 2024-03-19T02:56:49.304 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/34 file after the snapshot 2024-03-19T02:56:49.305 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/35 file after the snapshot 2024-03-19T02:56:49.306 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/36 file after the snapshot 2024-03-19T02:56:49.308 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/37 file after the snapshot 2024-03-19T02:56:49.309 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/38 file after the snapshot 2024-03-19T02:56:49.311 INFO:tasks.workunit.client.0.smithi092.stdout: create 181/39 file after the snapshot 2024-03-19T02:56:49.313 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/.snap/snap-subdir-test 2024-03-19T02:56:49.361 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/30 file after the snapshot 2024-03-19T02:56:49.645 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/31 file after the snapshot 2024-03-19T02:56:49.647 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/32 file after the snapshot 2024-03-19T02:56:49.649 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/33 file after the snapshot 2024-03-19T02:56:49.651 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/34 file after the snapshot 2024-03-19T02:56:49.653 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/35 file after the snapshot 2024-03-19T02:56:49.655 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/36 file after the snapshot 2024-03-19T02:56:49.657 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/37 file after the snapshot 2024-03-19T02:56:49.659 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/38 file after the snapshot 2024-03-19T02:56:49.661 INFO:tasks.workunit.client.0.smithi092.stdout: create 182/39 file after the snapshot 2024-03-19T02:56:49.663 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/.snap/snap-subdir-test 2024-03-19T02:56:49.704 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/30 file after the snapshot 2024-03-19T02:56:49.707 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/31 file after the snapshot 2024-03-19T02:56:49.709 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/32 file after the snapshot 2024-03-19T02:56:49.710 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/33 file after the snapshot 2024-03-19T02:56:49.711 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/34 file after the snapshot 2024-03-19T02:56:49.713 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/35 file after the snapshot 2024-03-19T02:56:49.714 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/36 file after the snapshot 2024-03-19T02:56:49.715 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/37 file after the snapshot 2024-03-19T02:56:49.717 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/38 file after the snapshot 2024-03-19T02:56:49.718 INFO:tasks.workunit.client.0.smithi092.stdout: create 183/39 file after the snapshot 2024-03-19T02:56:49.720 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/.snap/snap-subdir-test 2024-03-19T02:56:49.756 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/30 file after the snapshot 2024-03-19T02:56:49.764 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/31 file after the snapshot 2024-03-19T02:56:49.766 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/32 file after the snapshot 2024-03-19T02:56:49.768 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/33 file after the snapshot 2024-03-19T02:56:49.769 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/34 file after the snapshot 2024-03-19T02:56:49.770 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/35 file after the snapshot 2024-03-19T02:56:49.771 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/36 file after the snapshot 2024-03-19T02:56:49.773 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/37 file after the snapshot 2024-03-19T02:56:49.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/38 file after the snapshot 2024-03-19T02:56:49.775 INFO:tasks.workunit.client.0.smithi092.stdout: create 184/39 file after the snapshot 2024-03-19T02:56:49.777 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/.snap/snap-subdir-test 2024-03-19T02:56:49.820 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/30 file after the snapshot 2024-03-19T02:56:49.831 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/31 file after the snapshot 2024-03-19T02:56:49.832 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/32 file after the snapshot 2024-03-19T02:56:49.834 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/33 file after the snapshot 2024-03-19T02:56:49.836 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/34 file after the snapshot 2024-03-19T02:56:49.838 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/35 file after the snapshot 2024-03-19T02:56:49.840 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/36 file after the snapshot 2024-03-19T02:56:49.842 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/37 file after the snapshot 2024-03-19T02:56:49.844 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/38 file after the snapshot 2024-03-19T02:56:49.845 INFO:tasks.workunit.client.0.smithi092.stdout: create 185/39 file after the snapshot 2024-03-19T02:56:49.847 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/.snap/snap-subdir-test 2024-03-19T02:56:49.910 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/30 file after the snapshot 2024-03-19T02:56:50.205 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/31 file after the snapshot 2024-03-19T02:56:50.841 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/32 file after the snapshot 2024-03-19T02:56:50.842 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/33 file after the snapshot 2024-03-19T02:56:50.844 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/34 file after the snapshot 2024-03-19T02:56:50.846 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/35 file after the snapshot 2024-03-19T02:56:50.847 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/36 file after the snapshot 2024-03-19T02:56:50.849 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/37 file after the snapshot 2024-03-19T02:56:50.851 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/38 file after the snapshot 2024-03-19T02:56:50.852 INFO:tasks.workunit.client.0.smithi092.stdout: create 186/39 file after the snapshot 2024-03-19T02:56:50.854 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/.snap/snap-subdir-test 2024-03-19T02:56:50.918 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/30 file after the snapshot 2024-03-19T02:56:51.517 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/31 file after the snapshot 2024-03-19T02:56:51.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/32 file after the snapshot 2024-03-19T02:56:51.543 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/33 file after the snapshot 2024-03-19T02:56:52.048 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/34 file after the snapshot 2024-03-19T02:56:52.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:56:52.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.474+0000 7f2889000640 1 Processor -- start 2024-03-19T02:56:52.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.474+0000 7f2889000640 1 -- start start 2024-03-19T02:56:52.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.475+0000 7f2889000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 0x7f288406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.475+0000 7f2889000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2884071d90 con 0x7f2884151260 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.475+0000 7f2889000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28840c7df0 con 0x7f288406f270 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.475+0000 7f2889000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f28840c6c90 con 0x7f2884071190 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2882800640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f288406f270 0x7f2884161840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:39204/0 (socket says 172.21.15.92:39204) 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2881400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 0x7f288406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2882800640 1 -- 172.21.15.92:0/991637708 learned_addr learned my addr 172.21.15.92:0/991637708 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658983483 0 0) 0x7f28840c7df0 con 0x7f288406f270 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2864005730 con 0x7f288406f270 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881696354 0 0) 0x7f2884071d90 con 0x7f2884151260 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28840c7df0 con 0x7f2884151260 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2881400640 1 -- 172.21.15.92:0/991637708 >> v1:172.21.15.159:6789/0 conn(0x7f288406f270 legacy=0x7f2884161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2881400640 1 -- 172.21.15.92:0/991637708 >> v1:172.21.15.92:6789/0 conn(0x7f2884151260 legacy=0x7f28840c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.476+0000 7f2881400640 1 -- 172.21.15.92:0/991637708 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2884071620 con 0x7f2884071190 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.477+0000 7f2881400640 1 --2- 172.21.15.92:0/991637708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 0x7f288406ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f286c009dc0 tx=0x7f286c02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=746cd333578a270e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c039070 con 0x7f2884071190 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f286c02cd50 con 0x7f2884071190 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2880a00640 1 -- 172.21.15.92:0/991637708 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c034660 con 0x7f2884071190 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 -- 172.21.15.92:0/991637708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 msgr2=0x7f288406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 --2- 172.21.15.92:0/991637708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 0x7f288406ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f286c009dc0 tx=0x7f286c02bfa0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 -- 172.21.15.92:0/991637708 shutdown_connections 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 --2- 172.21.15.92:0/991637708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884071190 0x7f288406ed30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 -- 172.21.15.92:0/991637708 >> 172.21.15.92:0/991637708 conn(0x7f28840c8d90 msgr2=0x7f28840c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 -- 172.21.15.92:0/991637708 shutdown_connections 2024-03-19T02:56:52.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.478+0000 7f2889000640 1 -- 172.21.15.92:0/991637708 wait complete. 2024-03-19T02:56:52.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.480+0000 7f2889000640 1 Processor -- start 2024-03-19T02:56:52.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.480+0000 7f2889000640 1 -- start start 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 0x7f2884144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 0x7f28841efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f28840c6ee0 con 0x7f288406f270 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2884157d20 con 0x7f2884071190 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2889000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f28840c71c0 con 0x7f2884151260 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2881e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 0x7f2884144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2882800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2881400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 0x7f28841efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2882800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43872/0 (socket says 172.21.15.92:43872) 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2881400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 0x7f28841efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:43296/0 (socket says 172.21.15.92:43296) 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.481+0000 7f2882800640 1 -- 172.21.15.92:0/640016834 learned_addr learned my addr 172.21.15.92:0/640016834 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 -- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 msgr2=0x7f28841f0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 -- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 msgr2=0x7f28841efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 0x7f28841efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 -- 172.21.15.92:0/640016834 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2884157150 con 0x7f288406f270 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2882800640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2881e00640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 0x7f2884144290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f287400c3f0 tx=0x7f287400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874017070 con 0x7f288406f270 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28740052f0 con 0x7f288406f270 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2884157430 con 0x7f288406f270 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.482+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874003830 con 0x7f288406f270 2024-03-19T02:56:52.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.483+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2884154e30 con 0x7f288406f270 2024-03-19T02:56:52.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.484+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2884068de0 con 0x7f288406f270 2024-03-19T02:56:52.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.485+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f287401a070 con 0x7f288406f270 2024-03-19T02:56:52.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.485+0000 7f287a000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 0x7f286006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.486+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7f2874089d80 con 0x7f288406f270 2024-03-19T02:56:52.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.486+0000 7f2881400640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 0x7f286006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.486+0000 7f2881400640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 0x7f286006e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f286c036000 tx=0x7f286c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.490+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2884068de0 con 0x7f288406f270 2024-03-19T02:56:52.728 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.727+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28840684c0 con 0x7f288406f270 2024-03-19T02:56:52.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.728+0000 7f287a000640 1 -- 172.21.15.92:0/640016834 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f2874058170 con 0x7f288406f270 2024-03-19T02:56:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:56:52.730 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:56:52.730 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:56:52.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.730+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 msgr2=0x7f286006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.730+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 0x7f286006e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f286c036000 tx=0x7f286c030000 comp rx=0 tx=0).stop 2024-03-19T02:56:52.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 msgr2=0x7f2884144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 0x7f2884144290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f287400c3f0 tx=0x7f287400c8b0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 shutdown_connections 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f286006bbd0 0x7f286006e090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2884151260 0x7f28841f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2884071190 0x7f28841efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 --2- 172.21.15.92:0/640016834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f288406f270 0x7f2884144290 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 >> 172.21.15.92:0/640016834 conn(0x7f28840c8d90 msgr2=0x7f28840c4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 shutdown_connections 2024-03-19T02:56:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.731+0000 7f2889000640 1 -- 172.21.15.92:0/640016834 wait complete. 2024-03-19T02:56:52.752 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2024-03-19T02:56:52.752 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-19T02:56:52.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:56:52.762 INFO:tasks.ceph.mds.d:Stopped 2024-03-19T02:56:52.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:56:52.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.972+0000 7fb436e00640 1 Processor -- start 2024-03-19T02:56:52.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.972+0000 7fb436e00640 1 -- start start 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.973+0000 7fb436e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb430072380 0x7fb430071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb436e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb43015b740 con 0x7fb4300c60d0 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb436e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb43015ba20 con 0x7fb430154dc0 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb436e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb4300c8400 con 0x7fb430072380 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb42fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb430154dc0 0x7fb430071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:39218/0 (socket says 172.21.15.92:39218) 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb42fe00640 1 -- 172.21.15.92:0/2701296674 learned_addr learned my addr 172.21.15.92:0/2701296674 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:56:52.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb42f400640 1 --2- 172.21.15.92:0/2701296674 >> v2:172.21.15.92:3300/0 conn(0x7fb430072380 0x7fb430071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.975+0000 7fb42f400640 1 -- 172.21.15.92:0/2701296674 >> v1:172.21.15.159:6789/0 conn(0x7fb430154dc0 legacy=0x7fb430071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.975+0000 7fb42f400640 1 -- 172.21.15.92:0/2701296674 >> v1:172.21.15.92:6789/0 conn(0x7fb4300c60d0 legacy=0x7fb4300c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.974+0000 7fb42ea00640 1 -- 172.21.15.92:0/2701296674 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101620940 0 0) 0x7fb43015ba20 con 0x7fb430154dc0 2024-03-19T02:56:52.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.975+0000 7fb42f400640 1 -- 172.21.15.92:0/2701296674 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4300c86e0 con 0x7fb430072380 2024-03-19T02:56:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.976+0000 7fb42f400640 1 --2- 172.21.15.92:0/2701296674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430072380 0x7fb430071190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0097a0 tx=0x7fb41c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e28dcc420cacc14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.976+0000 7fb42ea00640 1 -- 172.21.15.92:0/2701296674 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c037070 con 0x7fb430072380 2024-03-19T02:56:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.976+0000 7fb42ea00640 1 -- 172.21.15.92:0/2701296674 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41c030710 con 0x7fb430072380 2024-03-19T02:56:52.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.976+0000 7fb42ea00640 1 -- 172.21.15.92:0/2701296674 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c030ae0 con 0x7fb430072380 2024-03-19T02:56:52.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 -- 172.21.15.92:0/2701296674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430072380 msgr2=0x7fb430071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 --2- 172.21.15.92:0/2701296674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430072380 0x7fb430071190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0097a0 tx=0x7fb41c02b710 comp rx=0 tx=0).stop 2024-03-19T02:56:52.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 -- 172.21.15.92:0/2701296674 shutdown_connections 2024-03-19T02:56:52.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 --2- 172.21.15.92:0/2701296674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430072380 0x7fb430071190 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 -- 172.21.15.92:0/2701296674 >> 172.21.15.92:0/2701296674 conn(0x7fb4300c8d90 msgr2=0x7fb4300c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:52.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 -- 172.21.15.92:0/2701296674 shutdown_connections 2024-03-19T02:56:52.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.977+0000 7fb436e00640 1 -- 172.21.15.92:0/2701296674 wait complete. 2024-03-19T02:56:52.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.979+0000 7fb436e00640 1 Processor -- start 2024-03-19T02:56:52.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.979+0000 7fb436e00640 1 -- start start 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 0x7fb430155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430154dc0 0x7fb430157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb4300c7520 con 0x7fb430154dc0 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb4300c73a0 con 0x7fb4300c60d0 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.980+0000 7fb436e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb4300c76a0 con 0x7fb430072380 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 0x7fb430155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 0x7fb430155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:43298/0 (socket says 172.21.15.92:43298) 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43886/0 (socket says 172.21.15.92:43886) 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42f400640 1 -- 172.21.15.92:0/2413056782 learned_addr learned my addr 172.21.15.92:0/2413056782 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:56:52.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb434e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430154dc0 0x7fb430157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 -- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 msgr2=0x7fb430155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 0x7fb430155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 -- 172.21.15.92:0/2413056782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430154dc0 msgr2=0x7fb430157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430154dc0 0x7fb430157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 -- 172.21.15.92:0/2413056782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43015af00 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.981+0000 7fb42fe00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb42000c3e0 tx=0x7fb42000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.982+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4200074e0 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.982+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4200047e0 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.982+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4301f1530 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.982+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420004c10 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.982+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43015ac20 con 0x7fb430072380 2024-03-19T02:56:52.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.983+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4300684c0 con 0x7fb430072380 2024-03-19T02:56:52.985 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.984+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb420011070 con 0x7fb430072380 2024-03-19T02:56:52.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.984+0000 7fb427e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 0x7fb41006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:52.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.985+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7fb42008a4e0 con 0x7fb430072380 2024-03-19T02:56:52.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.985+0000 7fb42f400640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 0x7fb41006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:52.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.985+0000 7fb42f400640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 0x7fb41006e090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb41c02b750 tx=0x7fb41c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:52.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:52.989+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb43015af00 con 0x7fb430072380 2024-03-19T02:56:53.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.228+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4300c7d00 con 0x7fb430072380 2024-03-19T02:56:53.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.229+0000 7fb427e00640 1 -- 172.21.15.92:0/2413056782 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fb4200588c0 con 0x7fb430072380 2024-03-19T02:56:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:56:53.231 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:56:53.231 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:56:53.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 msgr2=0x7fb41006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:53.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 0x7fb41006e090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb41c02b750 tx=0x7fb41c008030 comp rx=0 tx=0).stop 2024-03-19T02:56:53.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 msgr2=0x7fb430148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb42000c3e0 tx=0x7fb42000c8a0 comp rx=0 tx=0).stop 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 shutdown_connections 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb41006bbd0 0x7fb41006e090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb430154dc0 0x7fb430157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4300c60d0 0x7fb430155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:53.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 --2- 172.21.15.92:0/2413056782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb430072380 0x7fb430148890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:53.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.232+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 >> 172.21.15.92:0/2413056782 conn(0x7fb4300c8d90 msgr2=0x7fb430153060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:53.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.233+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 shutdown_connections 2024-03-19T02:56:53.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:53.233+0000 7fb436e00640 1 -- 172.21.15.92:0/2413056782 wait complete. 2024-03-19T02:56:53.254 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T02:56:55.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:56:55.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.472+0000 7fc908a00640 1 Processor -- start 2024-03-19T02:56:55.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.473+0000 7fc908a00640 1 -- start start 2024-03-19T02:56:55.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc908a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc9040cd1c0 0x7fc9040cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:55.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc908a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9040b3290 con 0x7fc904154de0 2024-03-19T02:56:55.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc908a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9040b2cf0 con 0x7fc9040b34c0 2024-03-19T02:56:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc908a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fc9040cdb30 con 0x7fc9040cd1c0 2024-03-19T02:56:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc902800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc904154de0 0x7fc9040cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:39000/0 (socket says 172.21.15.92:39000) 2024-03-19T02:56:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc902800640 1 -- 172.21.15.92:0/905598112 learned_addr learned my addr 172.21.15.92:0/905598112 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:56:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.474+0000 7fc903200640 1 --2- 172.21.15.92:0/905598112 >> v2:172.21.15.159:3300/0 conn(0x7fc9040cd1c0 0x7fc9040cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429558771 0 0) 0x7fc9040b3290 con 0x7fc904154de0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e8003520 con 0x7fc904154de0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252674351 0 0) 0x7fc9040b2cf0 con 0x7fc9040b34c0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9040b3290 con 0x7fc9040b34c0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc903200640 1 -- 172.21.15.92:0/905598112 >> v1:172.21.15.159:6789/0 conn(0x7fc9040b34c0 legacy=0x7fc90406bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc903200640 1 -- 172.21.15.92:0/905598112 >> v1:172.21.15.92:6789/0 conn(0x7fc904154de0 legacy=0x7fc9040cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.475+0000 7fc903200640 1 -- 172.21.15.92:0/905598112 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90415ef90 con 0x7fc9040cd1c0 2024-03-19T02:56:55.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.476+0000 7fc903200640 1 --2- 172.21.15.92:0/905598112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc9040cd5f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af1813ef6f2c1286 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.476+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc034070 con 0x7fc9040cd1c0 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.476+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc030610 con 0x7fc9040cd1c0 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.476+0000 7fc901400640 1 -- 172.21.15.92:0/905598112 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0309c0 con 0x7fc9040cd1c0 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 -- 172.21.15.92:0/905598112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 msgr2=0x7fc9040cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 --2- 172.21.15.92:0/905598112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc9040cd5f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b960 comp rx=0 tx=0).stop 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 -- 172.21.15.92:0/905598112 shutdown_connections 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 --2- 172.21.15.92:0/905598112 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc9040cd5f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 -- 172.21.15.92:0/905598112 >> 172.21.15.92:0/905598112 conn(0x7fc904067130 msgr2=0x7fc904067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 -- 172.21.15.92:0/905598112 shutdown_connections 2024-03-19T02:56:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.477+0000 7fc908a00640 1 -- 172.21.15.92:0/905598112 wait complete. 2024-03-19T02:56:55.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.479+0000 7fc908a00640 1 Processor -- start 2024-03-19T02:56:55.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.479+0000 7fc908a00640 1 -- start start 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc904159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc904154de0 0x7fc9041f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc90415e1d0 con 0x7fc9040b34c0 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc9040cccc0 con 0x7fc9040cd1c0 2024-03-19T02:56:55.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc908a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc90415e4b0 con 0x7fc904154de0 2024-03-19T02:56:55.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:55.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48478/0 (socket says 172.21.15.92:48478) 2024-03-19T02:56:55.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 -- 172.21.15.92:0/1883718738 learned_addr learned my addr 172.21.15.92:0/1883718738 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:56:55.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc903200640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc904154de0 0x7fc9041f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:55.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc901e00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc904159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 -- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc904154de0 msgr2=0x7fc9041f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc904154de0 0x7fc9041f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 -- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 msgr2=0x7fc904159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc904159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.480+0000 7fc902800640 1 -- 172.21.15.92:0/1883718738 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9040c6d50 con 0x7fc9040b34c0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc902800640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400cb50 tx=0x7fc8f400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4017070 con 0x7fc9040b34c0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f400ad40 con 0x7fc9040b34c0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9040c72b0 con 0x7fc9040b34c0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4012620 con 0x7fc9040b34c0 2024-03-19T02:56:55.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.481+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc904158ab0 con 0x7fc9040b34c0 2024-03-19T02:56:55.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.482+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9040cb920 con 0x7fc9040b34c0 2024-03-19T02:56:55.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.483+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc8f401a070 con 0x7fc9040b34c0 2024-03-19T02:56:55.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.484+0000 7fc8faa00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 0x7fc8e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:55.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.484+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7fc8f408a8f0 con 0x7fc9040b34c0 2024-03-19T02:56:55.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.484+0000 7fc901e00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 0x7fc8e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:55.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.485+0000 7fc901e00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 0x7fc8e406e0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0007fd0 tx=0x7fc8f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:55.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.489+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9040c6d50 con 0x7fc9040b34c0 2024-03-19T02:56:55.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.724+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc90404f580 con 0x7fc9040b34c0 2024-03-19T02:56:55.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.726+0000 7fc8faa00640 1 -- 172.21.15.92:0/1883718738 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fc8f4058c40 con 0x7fc9040b34c0 2024-03-19T02:56:55.728 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:56:55.728 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:56:55.728 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:56:55.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 msgr2=0x7fc8e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 0x7fc8e406e0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0007fd0 tx=0x7fc8f0009040 comp rx=0 tx=0).stop 2024-03-19T02:56:55.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 msgr2=0x7fc9040c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400cb50 tx=0x7fc8f400a430 comp rx=0 tx=0).stop 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 shutdown_connections 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc8e406bc00 0x7fc8e406e0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc904154de0 0x7fc9041f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9040cd1c0 0x7fc904159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 --2- 172.21.15.92:0/1883718738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9040b34c0 0x7fc9040c8050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 >> 172.21.15.92:0/1883718738 conn(0x7fc904067130 msgr2=0x7fc9041523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 shutdown_connections 2024-03-19T02:56:55.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:55.729+0000 7fc908a00640 1 -- 172.21.15.92:0/1883718738 wait complete. 2024-03-19T02:56:55.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T02:56:57.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:56:57.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.966+0000 7fd502c00640 1 Processor -- start 2024-03-19T02:56:57.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.966+0000 7fd502c00640 1 -- start start 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc0c8dc0 0x7fd4fc0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4fc155000 0x7fd4fc0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4fc0c9ef0 con 0x7fd4fc154c20 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd4fc15ae10 con 0x7fd4fc155000 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd4fc0ca190 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4fc155000 0x7fd4fc0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4fc155000 0x7fd4fc0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57670/0 (socket says 172.21.15.92:57670) 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd502200640 1 -- 172.21.15.92:0/2932559703 learned_addr learned my addr 172.21.15.92:0/2932559703 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.968+0000 7fd500e00640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc0c8dc0 0x7fd4fc0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.969+0000 7fd502200640 1 -- 172.21.15.92:0/2932559703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc0c8dc0 msgr2=0x7fd4fc0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.969+0000 7fd502200640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc0c8dc0 0x7fd4fc0cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.969+0000 7fd502200640 1 -- 172.21.15.92:0/2932559703 >> v1:172.21.15.92:6789/0 conn(0x7fd4fc154c20 legacy=0x7fd4fc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.969+0000 7fd502200640 1 -- 172.21.15.92:0/2932559703 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc0ca450 con 0x7fd4fc155000 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.970+0000 7fd502200640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc155000 0x7fd4fc0cb390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0cbe70 tx=0x7fd4f402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ab4f614b25383c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.970+0000 7fd4fbe00640 1 -- 172.21.15.92:0/2932559703 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f402ca30 con 0x7fd4fc155000 2024-03-19T02:56:57.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.970+0000 7fd4fbe00640 1 -- 172.21.15.92:0/2932559703 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f402cbd0 con 0x7fd4fc155000 2024-03-19T02:56:57.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.970+0000 7fd4fbe00640 1 -- 172.21.15.92:0/2932559703 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f40343f0 con 0x7fd4fc155000 2024-03-19T02:56:57.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 -- 172.21.15.92:0/2932559703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc155000 msgr2=0x7fd4fc0cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:57.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc155000 0x7fd4fc0cb390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0cbe70 tx=0x7fd4f402be90 comp rx=0 tx=0).stop 2024-03-19T02:56:57.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 -- 172.21.15.92:0/2932559703 shutdown_connections 2024-03-19T02:56:57.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc155000 0x7fd4fc0cb390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:57.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 --2- 172.21.15.92:0/2932559703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc0c8dc0 0x7fd4fc0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:57.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 -- 172.21.15.92:0/2932559703 >> 172.21.15.92:0/2932559703 conn(0x7fd4fc070d10 msgr2=0x7fd4fc071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:57.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 -- 172.21.15.92:0/2932559703 shutdown_connections 2024-03-19T02:56:57.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.971+0000 7fd502c00640 1 -- 172.21.15.92:0/2932559703 wait complete. 2024-03-19T02:56:57.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.973+0000 7fd502c00640 1 Processor -- start 2024-03-19T02:56:57.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.973+0000 7fd502c00640 1 -- start start 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.973+0000 7fd502c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc154c20 0x7fd4fc0c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4fc155000 0x7fd4fc1f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc15b9b0 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc0caa10 con 0x7fd4fc155000 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd4fc15bb30 con 0x7fd4fc154c20 2024-03-19T02:56:57.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57676/0 (socket says 172.21.15.92:57676) 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 -- 172.21.15.92:0/1996789152 learned_addr learned my addr 172.21.15.92:0/1996789152 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd502200640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4fc155000 0x7fd4fc1f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd500e00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc154c20 0x7fd4fc0c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 -- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc154c20 msgr2=0x7fd4fc0c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc154c20 0x7fd4fc0c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 -- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4fc155000 msgr2=0x7fd4fc1f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4fc155000 0x7fd4fc1f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.974+0000 7fd501800640 1 -- 172.21.15.92:0/1996789152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc1570d0 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd501800640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000b420 tx=0x7fd4f000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0015020 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f0011830 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0011c20 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4fc1573b0 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.975+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4fc1f4440 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.976+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4fc04f580 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.977+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd4f0018020 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.978+0000 7fd4f9600640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 0x7fd4dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:57.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.978+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7fd4f008a5f0 con 0x7fd4fc0c8dc0 2024-03-19T02:56:57.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.978+0000 7fd500e00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 0x7fd4dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:57.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.979+0000 7fd500e00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 0x7fd4dc06e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc1f50a0 tx=0x7fd4e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:57.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:57.983+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd4f001f030 con 0x7fd4fc0c8dc0 2024-03-19T02:56:58.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.218+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4fc1589a0 con 0x7fd4fc0c8dc0 2024-03-19T02:56:58.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.219+0000 7fd4f9600640 1 -- 172.21.15.92:0/1996789152 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fd4f0058940 con 0x7fd4fc0c8dc0 2024-03-19T02:56:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:56:58.222 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:56:58.222 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.222+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 msgr2=0x7fd4dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.222+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 0x7fd4dc06e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc1f50a0 tx=0x7fd4e8008040 comp rx=0 tx=0).stop 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.222+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 msgr2=0x7fd4fc156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.222+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000b420 tx=0x7fd4f000b8e0 comp rx=0 tx=0).stop 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 shutdown_connections 2024-03-19T02:56:58.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4dc06bbd0 0x7fd4dc06e090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4fc155000 0x7fd4fc1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4fc154c20 0x7fd4fc0c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 --2- 172.21.15.92:0/1996789152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4fc0c8dc0 0x7fd4fc156a40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 >> 172.21.15.92:0/1996789152 conn(0x7fd4fc070d10 msgr2=0x7fd4fc159180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 shutdown_connections 2024-03-19T02:56:58.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:56:58.223+0000 7fd502c00640 1 -- 172.21.15.92:0/1996789152 wait complete. 2024-03-19T02:56:58.244 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T02:56:59.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:56:59.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:57:00.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:00.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.464+0000 7f3530a00640 1 Processor -- start 2024-03-19T02:57:00.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.464+0000 7f3530a00640 1 -- start start 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.465+0000 7f3530a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f352c0c5b70 0x7f352c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.465+0000 7f3530a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f352c06d160 0x7f352c152c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.465+0000 7f3530a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352c0c6d00 con 0x7f352c0c5f50 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.465+0000 7f3530a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f352c06df20 con 0x7f352c0c5b70 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.465+0000 7f3530a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f352c06d610 con 0x7f352c06d160 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f352c0c5b70 0x7f352c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f3521e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f352c0c5f50 0x7f352c152520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:36928/0 (socket says 172.21.15.92:36928) 2024-03-19T02:57:00.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f352c0c5b70 0x7f352c15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57686/0 (socket says 172.21.15.92:57686) 2024-03-19T02:57:00.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352b200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f352c06d160 0x7f352c152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f3521e00640 1 -- 172.21.15.92:0/3792821047 learned_addr learned my addr 172.21.15.92:0/3792821047 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:00.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f3529e00640 1 -- 172.21.15.92:0/3792821047 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958857208 0 0) 0x7f352c0c6d00 con 0x7f352c0c5f50 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f3529e00640 1 -- 172.21.15.92:0/3792821047 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3510003520 con 0x7f352c0c5f50 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 -- 172.21.15.92:0/3792821047 >> v2:172.21.15.159:3300/0 conn(0x7f352c06d160 msgr2=0x7f352c152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 --2- 172.21.15.92:0/3792821047 >> v2:172.21.15.159:3300/0 conn(0x7f352c06d160 0x7f352c152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 -- 172.21.15.92:0/3792821047 >> v1:172.21.15.159:6789/0 conn(0x7f352c0c5f50 legacy=0x7f352c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.466+0000 7f352a800640 1 -- 172.21.15.92:0/3792821047 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c0c6fa0 con 0x7f352c0c5b70 2024-03-19T02:57:00.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f352a800640 1 --2- 172.21.15.92:0/3792821047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5b70 0x7f352c15be70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f352c0705b0 tx=0x7f351c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0ac95e4c2d636f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f3529e00640 1 -- 172.21.15.92:0/3792821047 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c034070 con 0x7f352c0c5b70 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f3529e00640 1 -- 172.21.15.92:0/3792821047 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351c030610 con 0x7f352c0c5b70 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f3529e00640 1 -- 172.21.15.92:0/3792821047 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0309e0 con 0x7f352c0c5b70 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f3530a00640 1 -- 172.21.15.92:0/3792821047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5b70 msgr2=0x7f352c15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.468+0000 7f3530a00640 1 --2- 172.21.15.92:0/3792821047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5b70 0x7f352c15be70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f352c0705b0 tx=0x7f351c02b960 comp rx=0 tx=0).stop 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 -- 172.21.15.92:0/3792821047 shutdown_connections 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 --2- 172.21.15.92:0/3792821047 >> v2:172.21.15.159:3300/0 conn(0x7f352c06d160 0x7f352c152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 --2- 172.21.15.92:0/3792821047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5b70 0x7f352c15be70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 -- 172.21.15.92:0/3792821047 >> 172.21.15.92:0/3792821047 conn(0x7f352c0cc1f0 msgr2=0x7f352c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 -- 172.21.15.92:0/3792821047 shutdown_connections 2024-03-19T02:57:00.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.469+0000 7f3530a00640 1 -- 172.21.15.92:0/3792821047 wait complete. 2024-03-19T02:57:00.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.470+0000 7f3530a00640 1 Processor -- start 2024-03-19T02:57:00.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 -- start start 2024-03-19T02:57:00.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 0x7f352c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 0x7f352c1f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f352c0729b0 con 0x7f352c0c5f50 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f352c06dbb0 con 0x7f352c0c5b70 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.471+0000 7f3530a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f352c06e3b0 con 0x7f352c06d160 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f352a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 0x7f352c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f352a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 0x7f352c153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:53582/0 (socket says 172.21.15.92:53582) 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f352b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 0x7f352c1f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46476/0 (socket says 172.21.15.92:46476) 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f352a800640 1 -- 172.21.15.92:0/433138200 learned_addr learned my addr 172.21.15.92:0/433138200 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 -- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 msgr2=0x7f352c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 0x7f352c153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 -- 172.21.15.92:0/433138200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 msgr2=0x7f352c1f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 0x7f352c1f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 -- 172.21.15.92:0/433138200 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c15bec0 con 0x7f352c0c5b70 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f352b200640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 0x7f352c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.472+0000 7f3521e00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3514000f60 tx=0x7f3514001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.473+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514017070 con 0x7f352c0c5b70 2024-03-19T02:57:00.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.473+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3514005ce0 con 0x7f352c0c5b70 2024-03-19T02:57:00.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.473+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514007310 con 0x7f352c0c5b70 2024-03-19T02:57:00.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.473+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352c15c1a0 con 0x7f352c0c5b70 2024-03-19T02:57:00.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.473+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352c1f1c60 con 0x7f352c0c5b70 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.474+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f352c04f580 con 0x7f352c0c5b70 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.475+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f351401a070 con 0x7f352c0c5b70 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.476+0000 7f3523400640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 0x7f350806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.476+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7f3514089df0 con 0x7f352c0c5b70 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.476+0000 7f352a800640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 0x7f350806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:00.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.477+0000 7f352a800640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 0x7f350806e0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f351c009560 tx=0x7f351c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:00.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.481+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f35140580b0 con 0x7f352c0c5b70 2024-03-19T02:57:00.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.718+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f352c1e5da0 con 0x7f352c0c5b70 2024-03-19T02:57:00.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.719+0000 7f3523400640 1 -- 172.21.15.92:0/433138200 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f351405c150 con 0x7f352c0c5b70 2024-03-19T02:57:00.723 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:00.723 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.721+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 msgr2=0x7f350806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.721+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 0x7f350806e0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f351c009560 tx=0x7f351c03b040 comp rx=0 tx=0).stop 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.721+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 msgr2=0x7f352c1e23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.721+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3514000f60 tx=0x7f3514001e70 comp rx=0 tx=0).stop 2024-03-19T02:57:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 shutdown_connections 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f350806bc00 0x7f350806e0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f352c0c5f50 0x7f352c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f352c0c5b70 0x7f352c1e23d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 --2- 172.21.15.92:0/433138200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f352c06d160 0x7f352c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 >> 172.21.15.92:0/433138200 conn(0x7f352c0cc1f0 msgr2=0x7f352c149ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 shutdown_connections 2024-03-19T02:57:00.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:00.722+0000 7f3530a00640 1 -- 172.21.15.92:0/433138200 wait complete. 2024-03-19T02:57:00.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T02:57:02.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:02.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.965+0000 7f1f4ec00640 1 Processor -- start 2024-03-19T02:57:02.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.965+0000 7f1f4ec00640 1 -- start start 2024-03-19T02:57:02.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.966+0000 7f1f4ec00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1f48152bf0 0x7f1f4806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.966+0000 7f1f4ec00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f480c6470 0x7f1f480c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.966+0000 7f1f4ec00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f480c9a80 con 0x7f1f481593d0 2024-03-19T02:57:02.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.966+0000 7f1f4ec00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1f480c6c90 con 0x7f1f480c6470 2024-03-19T02:57:02.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.966+0000 7f1f4ec00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1f48159910 con 0x7f1f48152bf0 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f4cc00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1f481593d0 0x7f1f480c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:38758/0 (socket says 172.21.15.92:38758) 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f4cc00640 1 -- 172.21.15.92:0/3382201720 learned_addr learned my addr 172.21.15.92:0/3382201720 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f47e00640 1 --2- 172.21.15.92:0/3382201720 >> v2:172.21.15.159:3300/0 conn(0x7f1f48152bf0 0x7f1f4806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f47400640 1 --2- 172.21.15.92:0/3382201720 >> v2:172.21.15.92:3300/0 conn(0x7f1f480c6470 0x7f1f480c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f46a00640 1 -- 172.21.15.92:0/3382201720 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586488322 0 0) 0x7f1f480c9a80 con 0x7f1f481593d0 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.967+0000 7f1f46a00640 1 -- 172.21.15.92:0/3382201720 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f2c003520 con 0x7f1f481593d0 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.968+0000 7f1f47e00640 1 -- 172.21.15.92:0/3382201720 >> v2:172.21.15.92:3300/0 conn(0x7f1f480c6470 msgr2=0x7f1f480c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:02.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.968+0000 7f1f47e00640 1 --2- 172.21.15.92:0/3382201720 >> v2:172.21.15.92:3300/0 conn(0x7f1f480c6470 0x7f1f480c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:02.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.968+0000 7f1f47e00640 1 -- 172.21.15.92:0/3382201720 >> v1:172.21.15.92:6789/0 conn(0x7f1f481593d0 legacy=0x7f1f480c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:02.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.968+0000 7f1f47e00640 1 -- 172.21.15.92:0/3382201720 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f480c8650 con 0x7f1f48152bf0 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f47e00640 1 --2- 172.21.15.92:0/3382201720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f48152bf0 0x7f1f4806eef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f48075180 tx=0x7f1f3802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54ddf8549e3e6393 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f46a00640 1 -- 172.21.15.92:0/3382201720 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380315a0 con 0x7f1f48152bf0 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f46a00640 1 -- 172.21.15.92:0/3382201720 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f38037070 con 0x7f1f48152bf0 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f46a00640 1 -- 172.21.15.92:0/3382201720 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38031960 con 0x7f1f48152bf0 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f4ec00640 1 -- 172.21.15.92:0/3382201720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f48152bf0 msgr2=0x7f1f4806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.969+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/3382201720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f48152bf0 0x7f1f4806eef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f48075180 tx=0x7f1f3802b840 comp rx=0 tx=0).stop 2024-03-19T02:57:02.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 -- 172.21.15.92:0/3382201720 shutdown_connections 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/3382201720 >> v2:172.21.15.92:3300/0 conn(0x7f1f480c6470 0x7f1f480c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/3382201720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f48152bf0 0x7f1f4806eef0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 -- 172.21.15.92:0/3382201720 >> 172.21.15.92:0/3382201720 conn(0x7f1f480c8d90 msgr2=0x7f1f480c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 -- 172.21.15.92:0/3382201720 shutdown_connections 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.970+0000 7f1f4ec00640 1 -- 172.21.15.92:0/3382201720 wait complete. 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.971+0000 7f1f4ec00640 1 Processor -- start 2024-03-19T02:57:02.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.971+0000 7f1f4ec00640 1 -- start start 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 0x7f1f481f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 0x7f1f48143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f481593d0 0x7f1f48143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4806ea60 con 0x7f1f481593d0 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4806e4a0 con 0x7f1f480c6470 2024-03-19T02:57:02.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.972+0000 7f1f4ec00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4806e6f0 con 0x7f1f48152bf0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 0x7f1f481f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 0x7f1f48143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 0x7f1f481f0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:46492/0 (socket says 172.21.15.92:46492) 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47e00640 1 -- 172.21.15.92:0/648393370 learned_addr learned my addr 172.21.15.92:0/648393370 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f4cc00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f481593d0 0x7f1f48143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 -- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 msgr2=0x7f1f481f0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 0x7f1f481f0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 -- 172.21.15.92:0/648393370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f481593d0 msgr2=0x7f1f48143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f481593d0 0x7f1f48143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 -- 172.21.15.92:0/648393370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f481f4af0 con 0x7f1f48152bf0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.973+0000 7f1f47400640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 0x7f1f48143a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400e880 tx=0x7f1f3400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.974+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34017070 con 0x7f1f48152bf0 2024-03-19T02:57:02.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.974+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f3400c940 con 0x7f1f48152bf0 2024-03-19T02:57:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.974+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400cd30 con 0x7f1f48152bf0 2024-03-19T02:57:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.974+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f48071190 con 0x7f1f48152bf0 2024-03-19T02:57:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.974+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f481f4810 con 0x7f1f48152bf0 2024-03-19T02:57:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.975+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f481522d0 con 0x7f1f48152bf0 2024-03-19T02:57:02.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.976+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1f3401a070 con 0x7f1f48152bf0 2024-03-19T02:57:02.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.976+0000 7f1f3fe00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:02.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.977+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5882+0+0 (secure 0 0 0) 0x7f1f3408aac0 con 0x7f1f48152bf0 2024-03-19T02:57:02.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.977+0000 7f1f47e00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:02.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.978+0000 7f1f47e00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 0x7f1f2806e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f3802b880 tx=0x7f1f3803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:02.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:02.982+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1f481522d0 con 0x7f1f48152bf0 2024-03-19T02:57:03.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.223+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f480684c0 con 0x7f1f48152bf0 2024-03-19T02:57:03.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.224+0000 7f1f3fe00640 1 -- 172.21.15.92:0/648393370 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f1f34058e10 con 0x7f1f48152bf0 2024-03-19T02:57:03.225 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:03.225 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:55:02.221279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.159:6839/1521694689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":1521694689},{"type":"v1","addr":"172.21.15.159:6839","nonce":1521694689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-19T02:57:03.225 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 8 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 msgr2=0x7f1f2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 0x7f1f2806e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f3802b880 tx=0x7f1f3803d040 comp rx=0 tx=0).stop 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 msgr2=0x7f1f48143a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 0x7f1f48143a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400e880 tx=0x7f1f3400ed40 comp rx=0 tx=0).stop 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 shutdown_connections 2024-03-19T02:57:03.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.227+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f481593d0 0x7f1f48143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.228+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f48152bf0 0x7f1f48143a40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.228+0000 7f1f4ec00640 1 --2- 172.21.15.92:0/648393370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f480c6470 0x7f1f481f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.228+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 >> 172.21.15.92:0/648393370 conn(0x7f1f480c8d90 msgr2=0x7f1f4814e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.228+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 shutdown_connections 2024-03-19T02:57:03.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:03.228+0000 7f1f4ec00640 1 -- 172.21.15.92:0/648393370 wait complete. 2024-03-19T02:57:03.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T02:57:05.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:05.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.463+0000 7f9cf4a00640 1 Processor -- start 2024-03-19T02:57:05.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.464+0000 7f9cf4a00640 1 -- start start 2024-03-19T02:57:05.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.465+0000 7f9cf4a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf0071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.465+0000 7f9cf4a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cf0072380 0x7f9cf00c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.465+0000 7f9cf4a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cf015b740 con 0x7f9cf00c60d0 2024-03-19T02:57:05.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.465+0000 7f9cf4a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9cf015ba20 con 0x7f9cf0072380 2024-03-19T02:57:05.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.465+0000 7f9cf4a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf00c8400 con 0x7f9cf0154dc0 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cf0072380 0x7f9cf00c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cf0072380 0x7f9cf00c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57732/0 (socket says 172.21.15.92:57732) 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cede00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9ce5400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9cf00c60d0 0x7f9cf0071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:36938/0 (socket says 172.21.15.92:36938) 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 -- 172.21.15.92:0/3057687020 learned_addr learned my addr 172.21.15.92:0/3057687020 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 -- 172.21.15.92:0/3057687020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 msgr2=0x7f9cf0071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf0071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 -- 172.21.15.92:0/3057687020 >> v1:172.21.15.159:6789/0 conn(0x7f9cf00c60d0 legacy=0x7f9cf0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cee800640 1 -- 172.21.15.92:0/3057687020 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf00c86e0 con 0x7f9cf0072380 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9cede00640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf0071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.466+0000 7f9ced400640 1 -- 172.21.15.92:0/3057687020 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456909499 0 0) 0x7f9cf015b740 con 0x7f9cf00c60d0 2024-03-19T02:57:05.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.467+0000 7f9cee800640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf0072380 0x7f9cf00c6c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009e50 tx=0x7f9ce802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6161ea93c158f545 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.467+0000 7f9ced400640 1 -- 172.21.15.92:0/3057687020 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce802ca80 con 0x7f9cf0072380 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.467+0000 7f9ced400640 1 -- 172.21.15.92:0/3057687020 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce802cc20 con 0x7f9cf0072380 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9ced400640 1 -- 172.21.15.92:0/3057687020 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80343f0 con 0x7f9cf0072380 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 -- 172.21.15.92:0/3057687020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf0072380 msgr2=0x7f9cf00c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf0072380 0x7f9cf00c6c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009e50 tx=0x7f9ce802bee0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 -- 172.21.15.92:0/3057687020 shutdown_connections 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf0072380 0x7f9cf00c6c90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/3057687020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf0071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 -- 172.21.15.92:0/3057687020 >> 172.21.15.92:0/3057687020 conn(0x7f9cf00c8d90 msgr2=0x7f9cf00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 -- 172.21.15.92:0/3057687020 shutdown_connections 2024-03-19T02:57:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.468+0000 7f9cf4a00640 1 -- 172.21.15.92:0/3057687020 wait complete. 2024-03-19T02:57:05.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.470+0000 7f9cf4a00640 1 Processor -- start 2024-03-19T02:57:05.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.470+0000 7f9cf4a00640 1 -- start start 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0072380 0x7f9cf014d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf01f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf00c7350 con 0x7f9cf00c60d0 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf00c71d0 con 0x7f9cf0072380 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cf4a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf00c74d0 con 0x7f9cf0154dc0 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9ce5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9ce5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57746/0 (socket says 172.21.15.92:57746) 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9ce5400640 1 -- 172.21.15.92:0/2926199419 learned_addr learned my addr 172.21.15.92:0/2926199419 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cee800640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf01f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.471+0000 7f9cede00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0072380 0x7f9cf014d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 -- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 msgr2=0x7f9cf01f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf01f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 -- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0072380 msgr2=0x7f9cf014d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0072380 0x7f9cf014d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 -- 172.21.15.92:0/2926199419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf014dec0 con 0x7f9cf00c60d0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce5400640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800c4d0 tx=0x7f9cd800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8018070 con 0x7f9cf00c60d0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd8007500 con 0x7f9cf00c60d0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8004a20 con 0x7f9cf00c60d0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.472+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf014d900 con 0x7f9cf00c60d0 2024-03-19T02:57:05.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.473+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf0157500 con 0x7f9cf00c60d0 2024-03-19T02:57:05.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.474+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf006cdf0 con 0x7f9cf00c60d0 2024-03-19T02:57:05.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.475+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9cd8004bc0 con 0x7f9cf00c60d0 2024-03-19T02:57:05.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.476+0000 7f9ce6a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 0x7f9ccc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.476+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f9cd8089d80 con 0x7f9cf00c60d0 2024-03-19T02:57:05.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.476+0000 7f9cede00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 0x7f9ccc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.477+0000 7f9cede00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 0x7f9ccc06e0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009560 tx=0x7f9ce0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.480+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9cf006cdf0 con 0x7f9cf00c60d0 2024-03-19T02:57:05.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.713+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cf0071510 con 0x7f9cf00c60d0 2024-03-19T02:57:05.714 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.714+0000 7f9ce6a00640 1 -- 172.21.15.92:0/2926199419 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6000 (secure 0 0 0) 0x7f9cf0071510 con 0x7f9cf00c60d0 2024-03-19T02:57:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:05.715 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:04.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:05.715 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 9 2024-03-19T02:57:05.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 msgr2=0x7f9ccc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 0x7f9ccc06e0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009560 tx=0x7f9ce0009190 comp rx=0 tx=0).stop 2024-03-19T02:57:05.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 msgr2=0x7f9cf015a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800c4d0 tx=0x7f9cd800c990 comp rx=0 tx=0).stop 2024-03-19T02:57:05.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 shutdown_connections 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9ccc06bc00 0x7f9ccc06e0c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cf0154dc0 0x7f9cf01f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf00c60d0 0x7f9cf015a940 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 --2- 172.21.15.92:0/2926199419 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cf0072380 0x7f9cf014d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 >> 172.21.15.92:0/2926199419 conn(0x7f9cf00c8d90 msgr2=0x7f9cf00c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 shutdown_connections 2024-03-19T02:57:05.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.717+0000 7f9cf4a00640 1 -- 172.21.15.92:0/2926199419 wait complete. 2024-03-19T02:57:05.734 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-03-19T02:57:05.734 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-19T02:57:05.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:05.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.941+0000 7ff6e8a00640 1 Processor -- start 2024-03-19T02:57:05.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.941+0000 7ff6e8a00640 1 -- start start 2024-03-19T02:57:05.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e8a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff6e40c6470 0x7ff6e40c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e8a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6e41593d0 0x7ff6e40c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e8a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e40c9a80 con 0x7ff6e4152bf0 2024-03-19T02:57:05.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e8a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e40c6c90 con 0x7ff6e40c6470 2024-03-19T02:57:05.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e8a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e4159910 con 0x7ff6e41593d0 2024-03-19T02:57:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e1400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff6e40c6470 0x7ff6e40c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6e41593d0 0x7ff6e40c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6e41593d0 0x7ff6e40c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51570/0 (socket says 172.21.15.92:51570) 2024-03-19T02:57:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 -- 172.21.15.92:0/1558753865 learned_addr learned my addr 172.21.15.92:0/1558753865 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e1e00640 1 --1- 172.21.15.92:0/1558753865 >> v1:172.21.15.159:6789/0 conn(0x7ff6e4152bf0 0x7ff6e406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:43680/0 (socket says 172.21.15.92:43680) 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 -- 172.21.15.92:0/1558753865 >> v2:172.21.15.92:3300/0 conn(0x7ff6e40c6470 msgr2=0x7ff6e40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 --2- 172.21.15.92:0/1558753865 >> v2:172.21.15.92:3300/0 conn(0x7ff6e40c6470 0x7ff6e40c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 -- 172.21.15.92:0/1558753865 >> v1:172.21.15.159:6789/0 conn(0x7ff6e4152bf0 legacy=0x7ff6e406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.943+0000 7ff6e2800640 1 -- 172.21.15.92:0/1558753865 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e40c8650 con 0x7ff6e41593d0 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.944+0000 7ff6e2800640 1 --2- 172.21.15.92:0/1558753865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c44c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40c5070 tx=0x7ff6dc02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7a975470e02a93a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e0a00640 1 -- 172.21.15.92:0/1558753865 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02ca80 con 0x7ff6e41593d0 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e0a00640 1 -- 172.21.15.92:0/1558753865 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6dc02cc20 con 0x7ff6e41593d0 2024-03-19T02:57:05.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e0a00640 1 -- 172.21.15.92:0/1558753865 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0343f0 con 0x7ff6e41593d0 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 -- 172.21.15.92:0/1558753865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 msgr2=0x7ff6e40c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/1558753865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c44c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40c5070 tx=0x7ff6dc02bee0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 -- 172.21.15.92:0/1558753865 shutdown_connections 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/1558753865 >> v2:172.21.15.92:3300/0 conn(0x7ff6e40c6470 0x7ff6e40c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/1558753865 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c44c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 -- 172.21.15.92:0/1558753865 >> 172.21.15.92:0/1558753865 conn(0x7ff6e40c8d90 msgr2=0x7ff6e40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 -- 172.21.15.92:0/1558753865 shutdown_connections 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.945+0000 7ff6e8a00640 1 -- 172.21.15.92:0/1558753865 wait complete. 2024-03-19T02:57:05.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.947+0000 7ff6e8a00640 1 Processor -- start 2024-03-19T02:57:05.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.947+0000 7ff6e8a00640 1 -- start start 2024-03-19T02:57:05.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e40c6470 0x7ff6e415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e406ea60 con 0x7ff6e4152bf0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e406e4a0 con 0x7ff6e41593d0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e8a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e406e6f0 con 0x7ff6e40c6470 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.948+0000 7ff6e1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49562/0 (socket says 172.21.15.92:49562) 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 -- 172.21.15.92:0/4026247699 learned_addr learned my addr 172.21.15.92:0/4026247699 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e2800640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1e00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e40c6470 0x7ff6e415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 -- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e40c6470 msgr2=0x7ff6e415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e40c6470 0x7ff6e415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 -- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 msgr2=0x7ff6e40c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 -- 172.21.15.92:0/4026247699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e41585d0 con 0x7ff6e4152bf0 2024-03-19T02:57:05.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6e1400640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000e9e0 tx=0x7ff6d000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.949+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0017070 con 0x7ff6e4152bf0 2024-03-19T02:57:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.950+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d000c9d0 con 0x7ff6e4152bf0 2024-03-19T02:57:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.950+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000cdc0 con 0x7ff6e4152bf0 2024-03-19T02:57:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.950+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e40cb830 con 0x7ff6e4152bf0 2024-03-19T02:57:05.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.950+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e41ee650 con 0x7ff6e4152bf0 2024-03-19T02:57:05.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.951+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e40684c0 con 0x7ff6e4152bf0 2024-03-19T02:57:05.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.952+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff6d001a070 con 0x7ff6e4152bf0 2024-03-19T02:57:05.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.952+0000 7ff6da000640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 0x7ff6c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.953+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7ff6d008aee0 con 0x7ff6e4152bf0 2024-03-19T02:57:05.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.953+0000 7ff6e1e00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 0x7ff6c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.953+0000 7ff6e1e00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 0x7ff6c406e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:05.957+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff6d0059280 con 0x7ff6e4152bf0 2024-03-19T02:57:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.175+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6e40c3b50 con 0x7ff6e4152bf0 2024-03-19T02:57:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.176+0000 7ff6da000640 1 -- 172.21.15.92:0/4026247699 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6000 (secure 0 0 0) 0x7ff6d005d320 con 0x7ff6e4152bf0 2024-03-19T02:57:06.178 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:06.178 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:04.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 9 2024-03-19T02:57:06.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 msgr2=0x7ff6c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 0x7ff6c406e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4000ed0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 msgr2=0x7ff6e41ef180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000e9e0 tx=0x7ff6d000eea0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 shutdown_connections 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff6c406bbd0 0x7ff6c406e090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6e41593d0 0x7ff6e40c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6e4152bf0 0x7ff6e41ef180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 --2- 172.21.15.92:0/4026247699 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6e40c6470 0x7ff6e415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 >> 172.21.15.92:0/4026247699 conn(0x7ff6e40c8d90 msgr2=0x7ff6e4151d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 shutdown_connections 2024-03-19T02:57:06.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:06.179+0000 7ff6e8a00640 1 -- 172.21.15.92:0/4026247699 wait complete. 2024-03-19T02:57:06.197 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4261": { "addr": "172.21.15.159:6837/2387435110", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 2387435110, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 2387435110, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4261, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T02:57:04.595791+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4261 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T02:57:08.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:08.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.414+0000 7f8b18e00640 1 Processor -- start 2024-03-19T02:57:08.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.415+0000 7f8b18e00640 1 -- start start 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.416+0000 7f8b18e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b14071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.416+0000 7f8b18e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8b140c60d0 0x7f8b14071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.416+0000 7f8b18e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1415b740 con 0x7f8b140721b0 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.416+0000 7f8b18e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f8b14072660 con 0x7f8b140c60d0 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.416+0000 7f8b18e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8b1415ba20 con 0x7f8b14154dc0 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b11e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b14071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b09400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8b140c60d0 0x7f8b14071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b12800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8b140721b0 0x7f8b140c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49642/0 (socket says 172.21.15.92:49642) 2024-03-19T02:57:08.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b12800640 1 -- 172.21.15.92:0/1005229404 learned_addr learned my addr 172.21.15.92:0/1005229404 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:08.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b11e00640 1 -- 172.21.15.92:0/1005229404 >> v2:172.21.15.92:3300/0 conn(0x7f8b140c60d0 msgr2=0x7f8b14071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:08.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b11e00640 1 --2- 172.21.15.92:0/1005229404 >> v2:172.21.15.92:3300/0 conn(0x7f8b140c60d0 0x7f8b14071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b11e00640 1 -- 172.21.15.92:0/1005229404 >> v1:172.21.15.92:6789/0 conn(0x7f8b140721b0 legacy=0x7f8b140c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.417+0000 7f8b11e00640 1 -- 172.21.15.92:0/1005229404 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b140c8400 con 0x7f8b14154dc0 2024-03-19T02:57:08.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.418+0000 7f8b11e00640 1 --2- 172.21.15.92:0/1005229404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b14071190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097a0 tx=0x7f8b0402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ada9422bcfe2f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:08.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.418+0000 7f8b11400640 1 -- 172.21.15.92:0/1005229404 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031780 con 0x7f8b14154dc0 2024-03-19T02:57:08.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.418+0000 7f8b11400640 1 -- 172.21.15.92:0/1005229404 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b04037070 con 0x7f8b14154dc0 2024-03-19T02:57:08.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.418+0000 7f8b11400640 1 -- 172.21.15.92:0/1005229404 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031b40 con 0x7f8b14154dc0 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 -- 172.21.15.92:0/1005229404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 msgr2=0x7f8b14071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 --2- 172.21.15.92:0/1005229404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b14071190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097a0 tx=0x7f8b0402b820 comp rx=0 tx=0).stop 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 -- 172.21.15.92:0/1005229404 shutdown_connections 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 --2- 172.21.15.92:0/1005229404 >> v2:172.21.15.92:3300/0 conn(0x7f8b140c60d0 0x7f8b14071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 --2- 172.21.15.92:0/1005229404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b14071190 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 -- 172.21.15.92:0/1005229404 >> 172.21.15.92:0/1005229404 conn(0x7f8b140c8d90 msgr2=0x7f8b140c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 -- 172.21.15.92:0/1005229404 shutdown_connections 2024-03-19T02:57:08.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.419+0000 7f8b18e00640 1 -- 172.21.15.92:0/1005229404 wait complete. 2024-03-19T02:57:08.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.421+0000 7f8b18e00640 1 Processor -- start 2024-03-19T02:57:08.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.421+0000 7f8b18e00640 1 -- start start 2024-03-19T02:57:08.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 0x7f8b1406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b1414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b140c73a0 con 0x7f8b140c60d0 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8b141551a0 con 0x7f8b140721b0 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b18e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8b140c7520 con 0x7f8b14154dc0 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b09400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b09400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49578/0 (socket says 172.21.15.92:49578) 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b09400640 1 -- 172.21.15.92:0/2910401678 learned_addr learned my addr 172.21.15.92:0/2910401678 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b11e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 0x7f8b1406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.422+0000 7f8b12800640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b1414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 -- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 msgr2=0x7f8b1414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b1414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 -- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 msgr2=0x7f8b1406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 0x7f8b1406f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 -- 172.21.15.92:0/2910401678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1414fe90 con 0x7f8b140c60d0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b11e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 0x7f8b1406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b09400640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9b0 tx=0x7f8afc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc017070 con 0x7f8b140c60d0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc00c9a0 con 0x7f8b140c60d0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b14155b10 con 0x7f8b140c60d0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00cd90 con 0x7f8b140c60d0 2024-03-19T02:57:08.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.423+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b141560d0 con 0x7f8b140c60d0 2024-03-19T02:57:08.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.425+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b14071c50 con 0x7f8b140c60d0 2024-03-19T02:57:08.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.426+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8afc01a070 con 0x7f8b140c60d0 2024-03-19T02:57:08.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.426+0000 7f8b0aa00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 0x7f8af006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:08.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.426+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f8afc08adb0 con 0x7f8b140c60d0 2024-03-19T02:57:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.427+0000 7f8b11e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 0x7f8af006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.427+0000 7f8b11e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 0x7f8af006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b04036000 tx=0x7f8b0403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:08.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.431+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b14071c50 con 0x7f8b140c60d0 2024-03-19T02:57:08.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.664+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b140684c0 con 0x7f8b140c60d0 2024-03-19T02:57:08.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.665+0000 7f8b0aa00640 1 -- 172.21.15.92:0/2910401678 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6000 (secure 0 0 0) 0x7f8afc05f030 con 0x7f8b140c60d0 2024-03-19T02:57:08.667 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:08.667 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:04.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:08.667 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 9 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 msgr2=0x7f8af006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 0x7f8af006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b04036000 tx=0x7f8b0403d040 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 msgr2=0x7f8b1414f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9b0 tx=0x7f8afc00ee70 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 shutdown_connections 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f8af006bc00 0x7f8af006e0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b14154dc0 0x7f8b1414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b140c60d0 0x7f8b1414f2e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 --2- 172.21.15.92:0/2910401678 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b140721b0 0x7f8b1406f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 >> 172.21.15.92:0/2910401678 conn(0x7f8b140c8d90 msgr2=0x7f8b140c78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 shutdown_connections 2024-03-19T02:57:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:08.668+0000 7f8b18e00640 1 -- 172.21.15.92:0/2910401678 wait complete. 2024-03-19T02:57:08.689 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:10.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:10.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.910+0000 7fef23e00640 1 Processor -- start 2024-03-19T02:57:10.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.910+0000 7fef23e00640 1 -- start start 2024-03-19T02:57:10.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef23e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef240c5b70 0x7fef2415be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef23e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fef240c5f50 0x7fef24152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef23e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef240c6d00 con 0x7fef2406d160 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef23e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fef2406df20 con 0x7fef240c5b70 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef23e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fef2406d610 con 0x7fef240c5f50 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef22a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef240c5b70 0x7fef2415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef22a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef240c5b70 0x7fef2415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49586/0 (socket says 172.21.15.92:49586) 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef22a00640 1 -- 172.21.15.92:0/2725828779 learned_addr learned my addr 172.21.15.92:0/2725828779 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.912+0000 7fef22000640 1 --2- 172.21.15.92:0/2725828779 >> v2:172.21.15.159:3300/0 conn(0x7fef240c5f50 0x7fef24152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.913+0000 7fef22a00640 1 -- 172.21.15.92:0/2725828779 >> v2:172.21.15.159:3300/0 conn(0x7fef240c5f50 msgr2=0x7fef24152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.913+0000 7fef22a00640 1 --2- 172.21.15.92:0/2725828779 >> v2:172.21.15.159:3300/0 conn(0x7fef240c5f50 0x7fef24152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.913+0000 7fef22a00640 1 -- 172.21.15.92:0/2725828779 >> v1:172.21.15.159:6789/0 conn(0x7fef2406d160 legacy=0x7fef24152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:10.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.913+0000 7fef22a00640 1 -- 172.21.15.92:0/2725828779 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef240c6fa0 con 0x7fef240c5b70 2024-03-19T02:57:10.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.914+0000 7fef22a00640 1 --2- 172.21.15.92:0/2725828779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef2415be70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef240cd2b0 tx=0x7fef1402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74f20cddac337 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:10.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.914+0000 7fef21600640 1 -- 172.21.15.92:0/2725828779 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1402f850 con 0x7fef240c5b70 2024-03-19T02:57:10.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.914+0000 7fef21600640 1 -- 172.21.15.92:0/2725828779 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef1402f9f0 con 0x7fef240c5b70 2024-03-19T02:57:10.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.914+0000 7fef21600640 1 -- 172.21.15.92:0/2725828779 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1402fdc0 con 0x7fef240c5b70 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 -- 172.21.15.92:0/2725828779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 msgr2=0x7fef2415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 --2- 172.21.15.92:0/2725828779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef2415be70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef240cd2b0 tx=0x7fef1402ba30 comp rx=0 tx=0).stop 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 -- 172.21.15.92:0/2725828779 shutdown_connections 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 --2- 172.21.15.92:0/2725828779 >> v2:172.21.15.159:3300/0 conn(0x7fef240c5f50 0x7fef24152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 --2- 172.21.15.92:0/2725828779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef2415be70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 -- 172.21.15.92:0/2725828779 >> 172.21.15.92:0/2725828779 conn(0x7fef240cc1f0 msgr2=0x7fef240cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 -- 172.21.15.92:0/2725828779 shutdown_connections 2024-03-19T02:57:10.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.915+0000 7fef23e00640 1 -- 172.21.15.92:0/2725828779 wait complete. 2024-03-19T02:57:10.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.917+0000 7fef23e00640 1 Processor -- start 2024-03-19T02:57:10.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.917+0000 7fef23e00640 1 -- start start 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef2406d160 0x7fef241f1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef240c5f50 0x7fef2415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fef240729b0 con 0x7fef240c5b70 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fef2406dbb0 con 0x7fef240c5f50 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fef2406e3b0 con 0x7fef2406d160 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef22000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef22000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49592/0 (socket says 172.21.15.92:49592) 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef22000640 1 -- 172.21.15.92:0/534156064 learned_addr learned my addr 172.21.15.92:0/534156064 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:10.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef22a00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef2406d160 0x7fef241f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.918+0000 7fef23400640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef240c5f50 0x7fef2415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 -- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef2406d160 msgr2=0x7fef241f1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef2406d160 0x7fef241f1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 -- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef240c5f50 msgr2=0x7fef2415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef240c5f50 0x7fef2415a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 -- 172.21.15.92:0/534156064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef24159f80 con 0x7fef240c5b70 2024-03-19T02:57:10.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef22000640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e8a0 tx=0x7fef1000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10017070 con 0x7fef240c5b70 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef1000c940 con 0x7fef240c5b70 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1000cd30 con 0x7fef240c5b70 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef24156100 con 0x7fef240c5b70 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.919+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef24153880 con 0x7fef240c5b70 2024-03-19T02:57:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.921+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef24149d90 con 0x7fef240c5b70 2024-03-19T02:57:10.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.921+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fef1001a070 con 0x7fef240c5b70 2024-03-19T02:57:10.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.922+0000 7fef1aa00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 0x7fef0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:10.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.922+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7fef1008aa60 con 0x7fef240c5b70 2024-03-19T02:57:10.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.922+0000 7fef22a00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 0x7fef0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:10.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.923+0000 7fef22a00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 0x7fef0406e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef14036000 tx=0x7fef14008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:10.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:10.927+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fef24149d90 con 0x7fef240c5b70 2024-03-19T02:57:11.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.166+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef240cbd00 con 0x7fef240c5b70 2024-03-19T02:57:11.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.167+0000 7fef1aa00640 1 -- 172.21.15.92:0/534156064 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6006 (secure 0 0 0) 0x7fef240cbd00 con 0x7fef240c5b70 2024-03-19T02:57:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:11.169 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:10.102796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:11.169 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 10 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 msgr2=0x7fef0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 0x7fef0406e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fef14036000 tx=0x7fef14008030 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 msgr2=0x7fef24159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e8a0 tx=0x7fef1000ed60 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 shutdown_connections 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fef0406bbd0 0x7fef0406e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef240c5f50 0x7fef2415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef240c5b70 0x7fef24159a40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 --2- 172.21.15.92:0/534156064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef2406d160 0x7fef241f1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:11.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.170+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 >> 172.21.15.92:0/534156064 conn(0x7fef240cc1f0 msgr2=0x7fef24149680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:11.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.171+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 shutdown_connections 2024-03-19T02:57:11.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:11.171+0000 7fef23e00640 1 -- 172.21.15.92:0/534156064 wait complete. 2024-03-19T02:57:11.194 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:13.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:13.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.412+0000 7f6350a00640 1 Processor -- start 2024-03-19T02:57:13.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.412+0000 7f6350a00640 1 -- start start 2024-03-19T02:57:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6350a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 0x7f634c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6350a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634c0b3290 con 0x7f634c0cd1c0 2024-03-19T02:57:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6350a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634c0b2cf0 con 0x7f634c0b34c0 2024-03-19T02:57:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6350a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f634c0cccc0 con 0x7f634c154de0 2024-03-19T02:57:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f634a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 0x7f634c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6341e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f634c0b34c0 0x7f634c06bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:43688/0 (socket says 172.21.15.92:43688) 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6341e00640 1 -- 172.21.15.92:0/1817505651 learned_addr learned my addr 172.21.15.92:0/1817505651 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f634a800640 1 -- 172.21.15.92:0/1817505651 >> v1:172.21.15.159:6789/0 conn(0x7f634c0b34c0 legacy=0x7f634c06bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.415+0000 7f634a800640 1 -- 172.21.15.92:0/1817505651 >> v1:172.21.15.92:6789/0 conn(0x7f634c0cd1c0 legacy=0x7f634c165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.414+0000 7f6349e00640 1 -- 172.21.15.92:0/1817505651 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487968599 0 0) 0x7f634c0b2cf0 con 0x7f634c0b34c0 2024-03-19T02:57:13.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.415+0000 7f634a800640 1 -- 172.21.15.92:0/1817505651 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c15ef90 con 0x7f634c154de0 2024-03-19T02:57:13.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.416+0000 7f634a800640 1 --2- 172.21.15.92:0/1817505651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 0x7f634c0cb170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f633c009dc0 tx=0x7f633c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=708627022cbf083e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:13.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.416+0000 7f6349e00640 1 -- 172.21.15.92:0/1817505651 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c039070 con 0x7f634c154de0 2024-03-19T02:57:13.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.416+0000 7f6349e00640 1 -- 172.21.15.92:0/1817505651 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633c02ca80 con 0x7f634c154de0 2024-03-19T02:57:13.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.416+0000 7f6349e00640 1 -- 172.21.15.92:0/1817505651 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c02ce50 con 0x7f634c154de0 2024-03-19T02:57:13.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 -- 172.21.15.92:0/1817505651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 msgr2=0x7f634c0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 --2- 172.21.15.92:0/1817505651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 0x7f634c0cb170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f633c009dc0 tx=0x7f633c02bd50 comp rx=0 tx=0).stop 2024-03-19T02:57:13.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 -- 172.21.15.92:0/1817505651 shutdown_connections 2024-03-19T02:57:13.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 --2- 172.21.15.92:0/1817505651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c154de0 0x7f634c0cb170 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 -- 172.21.15.92:0/1817505651 >> 172.21.15.92:0/1817505651 conn(0x7f634c067130 msgr2=0x7f634c067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:13.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 -- 172.21.15.92:0/1817505651 shutdown_connections 2024-03-19T02:57:13.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.417+0000 7f6350a00640 1 -- 172.21.15.92:0/1817505651 wait complete. 2024-03-19T02:57:13.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.418+0000 7f6350a00640 1 Processor -- start 2024-03-19T02:57:13.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 -- start start 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f634c0b34c0 0x7f634c14e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c0cd1c0 0x7f634c0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f634c15e4b0 con 0x7f634c154de0 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f634c15e1d0 con 0x7f634c0b34c0 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.419+0000 7f6350a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f634c15e790 con 0x7f634c0cd1c0 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49600/0 (socket says 172.21.15.92:49600) 2024-03-19T02:57:13.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 -- 172.21.15.92:0/3873709536 learned_addr learned my addr 172.21.15.92:0/3873709536 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634a800640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f634c0b34c0 0x7f634c14e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f6341e00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c0cd1c0 0x7f634c0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 -- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c0cd1c0 msgr2=0x7f634c0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c0cd1c0 0x7f634c0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 -- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f634c0b34c0 msgr2=0x7f634c14e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f634c0b34c0 0x7f634c14e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 -- 172.21.15.92:0/3873709536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c14f230 con 0x7f634c154de0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.420+0000 7f634b200640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f634400c7b0 tx=0x7f634400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.421+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344017070 con 0x7f634c154de0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.421+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6344005270 con 0x7f634c154de0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.421+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c0c7a90 con 0x7f634c154de0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.421+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634400a640 con 0x7f634c154de0 2024-03-19T02:57:13.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.421+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c0c8050 con 0x7f634c154de0 2024-03-19T02:57:13.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.422+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f634c0cf6d0 con 0x7f634c154de0 2024-03-19T02:57:13.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.423+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f634401a070 con 0x7f634c154de0 2024-03-19T02:57:13.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.424+0000 7f6343400640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 0x7f632806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:13.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.424+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f634408a070 con 0x7f634c154de0 2024-03-19T02:57:13.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.424+0000 7f634a800640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 0x7f632806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:13.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.425+0000 7f634a800640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 0x7f632806e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f633c036000 tx=0x7f633c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:13.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.428+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f634c0cf6d0 con 0x7f634c154de0 2024-03-19T02:57:13.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.667+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f634c04f580 con 0x7f634c154de0 2024-03-19T02:57:13.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.668+0000 7f6343400640 1 -- 172.21.15.92:0/3873709536 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6006 (secure 0 0 0) 0x7f6344058440 con 0x7f634c154de0 2024-03-19T02:57:13.670 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:13.671 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:10.102796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:13.671 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 10 2024-03-19T02:57:13.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.670+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 msgr2=0x7f632806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.670+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 0x7f632806e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f633c036000 tx=0x7f633c030000 comp rx=0 tx=0).stop 2024-03-19T02:57:13.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 msgr2=0x7f634c0c94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f634400c7b0 tx=0x7f634400cc70 comp rx=0 tx=0).stop 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 shutdown_connections 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f632806bbd0 0x7f632806e090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c154de0 0x7f634c0c94f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f634c0cd1c0 0x7f634c0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 --2- 172.21.15.92:0/3873709536 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f634c0b34c0 0x7f634c14e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 >> 172.21.15.92:0/3873709536 conn(0x7f634c067130 msgr2=0x7f634c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 shutdown_connections 2024-03-19T02:57:13.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:13.671+0000 7f6350a00640 1 -- 172.21.15.92:0/3873709536 wait complete. 2024-03-19T02:57:13.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:15.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:15.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.908+0000 7f2783200640 1 Processor -- start 2024-03-19T02:57:15.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.908+0000 7f2783200640 1 -- start start 2024-03-19T02:57:15.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2783200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f27840c5f50 0x7f2784152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2783200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784152a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2783200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2784152fa0 con 0x7f27840c5b70 2024-03-19T02:57:15.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2783200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f278415c580 con 0x7f27840c5f50 2024-03-19T02:57:15.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2783200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f278406df20 con 0x7f278406d160 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2781e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f27840c5b70 0x7f278415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48454/0 (socket says 172.21.15.92:48454) 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2781400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f27840c5f50 0x7f2784152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2781e00640 1 -- 172.21.15.92:0/1591511626 learned_addr learned my addr 172.21.15.92:0/1591511626 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2782800640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2780a00640 1 -- 172.21.15.92:0/1591511626 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533804154 0 0) 0x7f2784152fa0 con 0x7f27840c5b70 2024-03-19T02:57:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2781400640 1 -- 172.21.15.92:0/1591511626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 msgr2=0x7f2784152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:15.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.910+0000 7f2781400640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:15.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.911+0000 7f2781400640 1 -- 172.21.15.92:0/1591511626 >> v1:172.21.15.92:6789/0 conn(0x7f27840c5b70 legacy=0x7f278415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:15.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.911+0000 7f2781400640 1 -- 172.21.15.92:0/1591511626 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27840c6d00 con 0x7f27840c5f50 2024-03-19T02:57:15.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.912+0000 7f2781400640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5f50 0x7f2784152520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27700097a0 tx=0x7f277002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e541dc72fca879 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:15.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.912+0000 7f2780a00640 1 -- 172.21.15.92:0/1591511626 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770034070 con 0x7f27840c5f50 2024-03-19T02:57:15.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.912+0000 7f2780a00640 1 -- 172.21.15.92:0/1591511626 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2770030610 con 0x7f27840c5f50 2024-03-19T02:57:15.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.912+0000 7f2780a00640 1 -- 172.21.15.92:0/1591511626 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27700309e0 con 0x7f27840c5f50 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 -- 172.21.15.92:0/1591511626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5f50 msgr2=0x7f2784152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5f50 0x7f2784152520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27700097a0 tx=0x7f277002b960 comp rx=0 tx=0).stop 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 -- 172.21.15.92:0/1591511626 shutdown_connections 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 --2- 172.21.15.92:0/1591511626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5f50 0x7f2784152520 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 -- 172.21.15.92:0/1591511626 >> 172.21.15.92:0/1591511626 conn(0x7f27840cc1f0 msgr2=0x7f27840cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 -- 172.21.15.92:0/1591511626 shutdown_connections 2024-03-19T02:57:15.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.913+0000 7f2783200640 1 -- 172.21.15.92:0/1591511626 wait complete. 2024-03-19T02:57:15.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.915+0000 7f2783200640 1 Processor -- start 2024-03-19T02:57:15.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.915+0000 7f2783200640 1 -- start start 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27840c5f50 0x7f278415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f278406d8d0 con 0x7f27840c5b70 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f27840c6fa0 con 0x7f27840c5f50 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2783200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f278406dbb0 con 0x7f278406d160 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2781400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2781400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54542/0 (socket says 172.21.15.92:54542) 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2781400640 1 -- 172.21.15.92:0/826390365 learned_addr learned my addr 172.21.15.92:0/826390365 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2781e00640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.916+0000 7f2782800640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27840c5f50 0x7f278415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 -- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 msgr2=0x7f2784150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:15.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:15.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 -- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27840c5f50 msgr2=0x7f278415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:15.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27840c5f50 0x7f278415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:15.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 -- 172.21.15.92:0/826390365 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27840ca320 con 0x7f27840c5b70 2024-03-19T02:57:15.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2781400640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f27700098c0 tx=0x7f27700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770034040 con 0x7f27840c5b70 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277003aaa0 con 0x7f27840c5b70 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27700443f0 con 0x7f27840c5b70 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27841f10e0 con 0x7f27840c5b70 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.917+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27840c5200 con 0x7f27840c5b70 2024-03-19T02:57:15.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.919+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278404f580 con 0x7f27840c5b70 2024-03-19T02:57:15.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.920+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f277004b060 con 0x7f27840c5b70 2024-03-19T02:57:15.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.921+0000 7f277a000640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 0x7f276406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:15.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.921+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f27700ac750 con 0x7f27840c5b70 2024-03-19T02:57:15.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.922+0000 7f2781e00640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 0x7f276406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:15.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.922+0000 7f2781e00640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 0x7f276406e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2774000f60 tx=0x7f2774007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:15.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:15.926+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f277003f070 con 0x7f27840c5b70 2024-03-19T02:57:16.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.157+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27840ca8e0 con 0x7f27840c5b70 2024-03-19T02:57:16.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.159+0000 7f277a000640 1 -- 172.21.15.92:0/826390365 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6006 (secure 0 0 0) 0x7f277007ab00 con 0x7f27840c5b70 2024-03-19T02:57:16.160 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:16.160 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:10.102796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:16.160 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 10 2024-03-19T02:57:16.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 msgr2=0x7f276406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:16.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 0x7f276406e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2774000f60 tx=0x7f2774007d50 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 msgr2=0x7f278415ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f27700098c0 tx=0x7f27700046c0 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 shutdown_connections 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f276406bbd0 0x7f276406e090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27840c5f50 0x7f278415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27840c5b70 0x7f278415ada0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 --2- 172.21.15.92:0/826390365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f278406d160 0x7f2784150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:16.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 >> 172.21.15.92:0/826390365 conn(0x7f27840cc1f0 msgr2=0x7f27841522f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:16.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.161+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 shutdown_connections 2024-03-19T02:57:16.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:16.162+0000 7f2783200640 1 -- 172.21.15.92:0/826390365 wait complete. 2024-03-19T02:57:16.179 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:18.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:18.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.390+0000 7f3c4e800640 1 Processor -- start 2024-03-19T02:57:18.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.390+0000 7f3c4e800640 1 -- start start 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3c48152bf0 0x7f3c4806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 0x7f3c480c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3c480c6c90 con 0x7f3c480c6470 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f3c48159910 con 0x7f3c48152bf0 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c4e800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4806e4a0 con 0x7f3c481593d0 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c3ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 0x7f3c480c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c47e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c47400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3c48152bf0 0x7f3c4806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c47e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:38970/0 (socket says 172.21.15.92:38970) 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c47e00640 1 -- 172.21.15.92:0/1527808250 learned_addr learned my addr 172.21.15.92:0/1527808250 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.392+0000 7f3c3ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 0x7f3c480c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54554/0 (socket says 172.21.15.92:54554) 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47400640 1 -- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 msgr2=0x7f3c480c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47400640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47400640 1 -- 172.21.15.92:0/1527808250 >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 msgr2=0x7f3c480c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47400640 1 --2- 172.21.15.92:0/1527808250 >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 0x7f3c480c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47400640 1 -- 172.21.15.92:0/1527808250 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c480c9a80 con 0x7f3c48152bf0 2024-03-19T02:57:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.393+0000 7f3c47e00640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:18.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.394+0000 7f3c47400640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c48152bf0 0x7f3c4806eef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c48075180 tx=0x7f3c3802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ffba4b7641cdcd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:18.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.394+0000 7f3c46a00640 1 -- 172.21.15.92:0/1527808250 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38038070 con 0x7f3c48152bf0 2024-03-19T02:57:18.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.395+0000 7f3c46a00640 1 -- 172.21.15.92:0/1527808250 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c38031710 con 0x7f3c48152bf0 2024-03-19T02:57:18.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.395+0000 7f3c46a00640 1 -- 172.21.15.92:0/1527808250 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38031ae0 con 0x7f3c48152bf0 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.395+0000 7f3c4e800640 1 -- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c48152bf0 msgr2=0x7f3c4806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.395+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c48152bf0 0x7f3c4806eef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c48075180 tx=0x7f3c3802b7b0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 -- 172.21.15.92:0/1527808250 shutdown_connections 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c481593d0 0x7f3c480c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1527808250 >> v2:172.21.15.92:3300/0 conn(0x7f3c480c6470 0x7f3c480c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1527808250 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c48152bf0 0x7f3c4806eef0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 -- 172.21.15.92:0/1527808250 >> 172.21.15.92:0/1527808250 conn(0x7f3c480c8d90 msgr2=0x7f3c480c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 -- 172.21.15.92:0/1527808250 shutdown_connections 2024-03-19T02:57:18.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.396+0000 7f3c4e800640 1 -- 172.21.15.92:0/1527808250 wait complete. 2024-03-19T02:57:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.398+0000 7f3c4e800640 1 Processor -- start 2024-03-19T02:57:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.398+0000 7f3c4e800640 1 -- start start 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c48152bf0 0x7f3c48143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c481593d0 0x7f3c48147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3c480c8930 con 0x7f3c480c6470 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4806ea60 con 0x7f3c481593d0 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c4e800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4806e6f0 con 0x7f3c48152bf0 2024-03-19T02:57:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c47400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c47400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54570/0 (socket says 172.21.15.92:54570) 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.399+0000 7f3c47400640 1 -- 172.21.15.92:0/1035665293 learned_addr learned my addr 172.21.15.92:0/1035665293 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c3ea00640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c48152bf0 0x7f3c48143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47e00640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c481593d0 0x7f3c48147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 -- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c48152bf0 msgr2=0x7f3c48143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c48152bf0 0x7f3c48143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 -- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c481593d0 msgr2=0x7f3c48147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c481593d0 0x7f3c48147c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 -- 172.21.15.92:0/1035665293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c48153760 con 0x7f3c480c6470 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.400+0000 7f3c47400640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3c38000c00 tx=0x7f3c380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.401+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38042070 con 0x7f3c480c6470 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.401+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3803d870 con 0x7f3c480c6470 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.401+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38038040 con 0x7f3c480c6470 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.401+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c48147740 con 0x7f3c480c6470 2024-03-19T02:57:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.401+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c481f46b0 con 0x7f3c480c6470 2024-03-19T02:57:18.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.402+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c480684c0 con 0x7f3c480c6470 2024-03-19T02:57:18.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.403+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3c3803b070 con 0x7f3c480c6470 2024-03-19T02:57:18.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.404+0000 7f3c3fe00640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 0x7f3c2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:18.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.404+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f3c380ac360 con 0x7f3c480c6470 2024-03-19T02:57:18.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.404+0000 7f3c3ea00640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 0x7f3c2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:18.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.405+0000 7f3c3ea00640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 0x7f3c2406e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cc260 tx=0x7f3c30009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:18.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.408+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3c3807a670 con 0x7f3c480c6470 2024-03-19T02:57:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.644+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c48147050 con 0x7f3c480c6470 2024-03-19T02:57:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.645+0000 7f3c3fe00640 1 -- 172.21.15.92:0/1035665293 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6003 (secure 0 0 0) 0x7f3c3807e710 con 0x7f3c480c6470 2024-03-19T02:57:18.647 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:18.647 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:16.042552+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:18.648 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 11 2024-03-19T02:57:18.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 msgr2=0x7f3c2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 0x7f3c2406e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cc260 tx=0x7f3c30009190 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 msgr2=0x7f3c48072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3c38000c00 tx=0x7f3c380046c0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 shutdown_connections 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f3c2406bbd0 0x7f3c2406e090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c481593d0 0x7f3c48147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c48152bf0 0x7f3c48143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 --2- 172.21.15.92:0/1035665293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c480c6470 0x7f3c48072d10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 >> 172.21.15.92:0/1035665293 conn(0x7f3c480c8d90 msgr2=0x7f3c480c77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 shutdown_connections 2024-03-19T02:57:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:18.648+0000 7f3c4e800640 1 -- 172.21.15.92:0/1035665293 wait complete. 2024-03-19T02:57:18.669 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4261": { "addr": "172.21.15.159:6837/2387435110", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 2387435110, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 2387435110, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4261, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T02:57:16.042552+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4261 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T02:57:20.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:20.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.888+0000 7fdbf9c00640 1 Processor -- start 2024-03-19T02:57:20.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.888+0000 7fdbf9c00640 1 -- start start 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.889+0000 7fdbf9c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdbf4151ce0 0x7fdbf4158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.889+0000 7fdbf9c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdbf40c7da0 0x7fdbf40c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.889+0000 7fdbf9c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf40c3b80 con 0x7fdbf4151900 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.889+0000 7fdbf9c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf4158860 con 0x7fdbf40c7da0 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.889+0000 7fdbf9c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf4072e70 con 0x7fdbf4151ce0 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdbf40c7da0 0x7fdbf40c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf2a00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdbf4151900 0x7fdbf40723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54766/0 (socket says 172.21.15.92:54766) 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf2a00640 1 -- 172.21.15.92:0/2248862980 learned_addr learned my addr 172.21.15.92:0/2248862980 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:20.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbea000640 1 --2- 172.21.15.92:0/2248862980 >> v2:172.21.15.159:3300/0 conn(0x7fdbf4151ce0 0x7fdbf4158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf2000640 1 -- 172.21.15.92:0/2248862980 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527081379 0 0) 0x7fdbf40c3b80 con 0x7fdbf4151900 2024-03-19T02:57:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf2000640 1 -- 172.21.15.92:0/2248862980 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd8003520 con 0x7fdbf4151900 2024-03-19T02:57:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf3400640 1 -- 172.21.15.92:0/2248862980 >> v2:172.21.15.159:3300/0 conn(0x7fdbf4151ce0 msgr2=0x7fdbf4158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf3400640 1 --2- 172.21.15.92:0/2248862980 >> v2:172.21.15.159:3300/0 conn(0x7fdbf4151ce0 0x7fdbf4158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf3400640 1 -- 172.21.15.92:0/2248862980 >> v1:172.21.15.159:6789/0 conn(0x7fdbf4151900 legacy=0x7fdbf40723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:20.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.890+0000 7fdbf3400640 1 -- 172.21.15.92:0/2248862980 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf4073150 con 0x7fdbf40c7da0 2024-03-19T02:57:20.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.892+0000 7fdbf3400640 1 --2- 172.21.15.92:0/2248862980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf40c7da0 0x7fdbf40c81f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40c4a30 tx=0x7fdbec02e130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8ecf962b85871e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:20.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.892+0000 7fdbf2000640 1 -- 172.21.15.92:0/2248862980 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec03b070 con 0x7fdbf40c7da0 2024-03-19T02:57:20.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.892+0000 7fdbf2000640 1 -- 172.21.15.92:0/2248862980 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbec036440 con 0x7fdbf40c7da0 2024-03-19T02:57:20.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.892+0000 7fdbf2000640 1 -- 172.21.15.92:0/2248862980 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec036810 con 0x7fdbf40c7da0 2024-03-19T02:57:20.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2248862980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf40c7da0 msgr2=0x7fdbf40c81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:20.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2248862980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf40c7da0 0x7fdbf40c81f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40c4a30 tx=0x7fdbec02e130 comp rx=0 tx=0).stop 2024-03-19T02:57:20.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2248862980 shutdown_connections 2024-03-19T02:57:20.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2248862980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf40c7da0 0x7fdbf40c81f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2248862980 >> v2:172.21.15.159:3300/0 conn(0x7fdbf4151ce0 0x7fdbf4158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2248862980 >> 172.21.15.92:0/2248862980 conn(0x7fdbf40c8d90 msgr2=0x7fdbf40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2248862980 shutdown_connections 2024-03-19T02:57:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.893+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2248862980 wait complete. 2024-03-19T02:57:20.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.894+0000 7fdbf9c00640 1 Processor -- start 2024-03-19T02:57:20.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 -- start start 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdbf40c7da0 0x7fdbf4152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdbf4151ce0 0x7fdbf414a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf40c9af0 con 0x7fdbf4151900 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf4071eb0 con 0x7fdbf40c7da0 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.895+0000 7fdbf9c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf40c9c70 con 0x7fdbf4151ce0 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54576/0 (socket says 172.21.15.92:54576) 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 -- 172.21.15.92:0/2818913216 learned_addr learned my addr 172.21.15.92:0/2818913216 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:20.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbf3400640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdbf4151ce0 0x7fdbf414a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbf2a00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdbf40c7da0 0x7fdbf4152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 -- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdbf4151ce0 msgr2=0x7fdbf414a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdbf4151ce0 0x7fdbf414a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 -- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdbf40c7da0 msgr2=0x7fdbf4152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdbf40c7da0 0x7fdbf4152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 -- 172.21.15.92:0/2818913216 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf4152a30 con 0x7fdbf4151900 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.896+0000 7fdbea000640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00efc0 tx=0x7fdbdc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.897+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc018020 con 0x7fdbf4151900 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.897+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbdc014480 con 0x7fdbf4151900 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.897+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc014850 con 0x7fdbf4151900 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.897+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf406f300 con 0x7fdbf4151900 2024-03-19T02:57:20.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.897+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf406ed40 con 0x7fdbf4151900 2024-03-19T02:57:20.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.898+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf40684c0 con 0x7fdbf4151900 2024-03-19T02:57:20.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.899+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdbdc005d10 con 0x7fdbf4151900 2024-03-19T02:57:20.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.900+0000 7fdbeb400640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 0x7fdbd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:20.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.900+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7fdbdc08b6d0 con 0x7fdbf4151900 2024-03-19T02:57:20.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.901+0000 7fdbf2a00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 0x7fdbd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:20.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.901+0000 7fdbf2a00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 0x7fdbd006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40011e0 tx=0x7fdbe4007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:20.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:20.905+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdbdc022070 con 0x7fdbf4151900 2024-03-19T02:57:21.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.141+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbf40caa20 con 0x7fdbf4151900 2024-03-19T02:57:21.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.142+0000 7fdbeb400640 1 -- 172.21.15.92:0/2818913216 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6003 (secure 0 0 0) 0x7fdbdc010040 con 0x7fdbf4151900 2024-03-19T02:57:21.144 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:21.144 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:16.042552+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:21.144 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 11 2024-03-19T02:57:21.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 msgr2=0x7fdbd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:21.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 0x7fdbd006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40011e0 tx=0x7fdbe4007d30 comp rx=0 tx=0).stop 2024-03-19T02:57:21.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 msgr2=0x7fdbf4157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00efc0 tx=0x7fdbdc00c490 comp rx=0 tx=0).stop 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 shutdown_connections 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdbd006bbd0 0x7fdbd006e090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdbf4151ce0 0x7fdbf414a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4151900 0x7fdbf4157340 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 --2- 172.21.15.92:0/2818913216 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdbf40c7da0 0x7fdbf4152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 >> 172.21.15.92:0/2818913216 conn(0x7fdbf40c8d90 msgr2=0x7fdbf4074740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 shutdown_connections 2024-03-19T02:57:21.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:21.145+0000 7fdbf9c00640 1 -- 172.21.15.92:0/2818913216 wait complete. 2024-03-19T02:57:21.166 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:23.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:23.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.381+0000 7fb0a0200640 1 Processor -- start 2024-03-19T02:57:23.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.381+0000 7fb0a0200640 1 -- start start 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb0a0200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 0x7fb09806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb0a0200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb0981593d0 0x7fb0980c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb0a0200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0980c9a80 con 0x7fb0980c6470 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb0a0200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb0980c6c90 con 0x7fb0981593d0 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb0a0200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb098159910 con 0x7fb098152bf0 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb09d800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 0x7fb09806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb09ce00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb0980c6470 0x7fb0980c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54774/0 (socket says 172.21.15.92:54774) 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb09ce00640 1 -- 172.21.15.92:0/3082242104 learned_addr learned my addr 172.21.15.92:0/3082242104 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.383+0000 7fb09e200640 1 --2- 172.21.15.92:0/3082242104 >> v2:172.21.15.92:3300/0 conn(0x7fb0981593d0 0x7fb0980c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb097e00640 1 -- 172.21.15.92:0/3082242104 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004180633 0 0) 0x7fb0980c9a80 con 0x7fb0980c6470 2024-03-19T02:57:23.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb097e00640 1 -- 172.21.15.92:0/3082242104 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb080003520 con 0x7fb0980c6470 2024-03-19T02:57:23.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb09d800640 1 -- 172.21.15.92:0/3082242104 >> v2:172.21.15.92:3300/0 conn(0x7fb0981593d0 msgr2=0x7fb0980c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:23.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb09d800640 1 --2- 172.21.15.92:0/3082242104 >> v2:172.21.15.92:3300/0 conn(0x7fb0981593d0 0x7fb0980c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb09d800640 1 -- 172.21.15.92:0/3082242104 >> v1:172.21.15.159:6789/0 conn(0x7fb0980c6470 legacy=0x7fb0980c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.384+0000 7fb09d800640 1 -- 172.21.15.92:0/3082242104 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980c8650 con 0x7fb098152bf0 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.385+0000 7fb09d800640 1 --2- 172.21.15.92:0/3082242104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 0x7fb09806eef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0097a0 tx=0x7fb08c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ab026f19ad9d774 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.385+0000 7fb097e00640 1 -- 172.21.15.92:0/3082242104 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c031780 con 0x7fb098152bf0 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.385+0000 7fb097e00640 1 -- 172.21.15.92:0/3082242104 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08c037070 con 0x7fb098152bf0 2024-03-19T02:57:23.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.385+0000 7fb097e00640 1 -- 172.21.15.92:0/3082242104 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c031b40 con 0x7fb098152bf0 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 -- 172.21.15.92:0/3082242104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 msgr2=0x7fb09806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 --2- 172.21.15.92:0/3082242104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 0x7fb09806eef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0097a0 tx=0x7fb08c02b820 comp rx=0 tx=0).stop 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 -- 172.21.15.92:0/3082242104 shutdown_connections 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 --2- 172.21.15.92:0/3082242104 >> v2:172.21.15.92:3300/0 conn(0x7fb0981593d0 0x7fb0980c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 --2- 172.21.15.92:0/3082242104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb098152bf0 0x7fb09806eef0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 -- 172.21.15.92:0/3082242104 >> 172.21.15.92:0/3082242104 conn(0x7fb0980c8d90 msgr2=0x7fb0980c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 -- 172.21.15.92:0/3082242104 shutdown_connections 2024-03-19T02:57:23.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.386+0000 7fb0a0200640 1 -- 172.21.15.92:0/3082242104 wait complete. 2024-03-19T02:57:23.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.388+0000 7fb0a0200640 1 Processor -- start 2024-03-19T02:57:23.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.388+0000 7fb0a0200640 1 -- start start 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0980c6470 0x7fb09815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 0x7fb098150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb09806ea60 con 0x7fb0980c6470 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb09806e4a0 con 0x7fb098152bf0 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb0a0200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb09806e6f0 con 0x7fb0981593d0 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb09e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb09e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:39014/0 (socket says 172.21.15.92:39014) 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb09e200640 1 -- 172.21.15.92:0/337457713 learned_addr learned my addr 172.21.15.92:0/337457713 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:23.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb09ce00640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 0x7fb098150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.389+0000 7fb09d800640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0980c6470 0x7fb09815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 -- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 msgr2=0x7fb0981f0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 -- 172.21.15.92:0/337457713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0980c6470 msgr2=0x7fb09815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0980c6470 0x7fb09815a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 -- 172.21.15.92:0/337457713 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09815ac60 con 0x7fb098152bf0 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09e200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:23.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb09ce00640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 0x7fb098150680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c400 tx=0x7fb08400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084017070 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb084005ce0 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840072f0 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09815b430 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.390+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb098146e90 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.391+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0980684c0 con 0x7fb098152bf0 2024-03-19T02:57:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.392+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb08401a070 con 0x7fb098152bf0 2024-03-19T02:57:23.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.393+0000 7fb095600640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:23.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.393+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7fb084089f90 con 0x7fb098152bf0 2024-03-19T02:57:23.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.394+0000 7fb09d800640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:23.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.394+0000 7fb09d800640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 0x7fb07806e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb08c036000 tx=0x7fb08c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:23.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.398+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb084058210 con 0x7fb098152bf0 2024-03-19T02:57:23.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.636+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0980c3b50 con 0x7fb098152bf0 2024-03-19T02:57:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.637+0000 7fb095600640 1 -- 172.21.15.92:0/337457713 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6009 (secure 0 0 0) 0x7fb08405c2b0 con 0x7fb098152bf0 2024-03-19T02:57:23.639 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:23.639 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:22.102045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 12 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 msgr2=0x7fb07806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 0x7fb07806e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb08c036000 tx=0x7fb08c03d040 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 msgr2=0x7fb098150680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 0x7fb098150680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c400 tx=0x7fb08400c8c0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 shutdown_connections 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0981593d0 0x7fb0981f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb098152bf0 0x7fb098150680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 --2- 172.21.15.92:0/337457713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0980c6470 0x7fb09815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:23.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 >> 172.21.15.92:0/337457713 conn(0x7fb0980c8d90 msgr2=0x7fb09814f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:23.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 shutdown_connections 2024-03-19T02:57:23.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:23.641+0000 7fb0a0200640 1 -- 172.21.15.92:0/337457713 wait complete. 2024-03-19T02:57:23.662 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:25.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:25.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.879+0000 7f82eae00640 1 Processor -- start 2024-03-19T02:57:25.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.879+0000 7f82eae00640 1 -- start start 2024-03-19T02:57:25.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82eae00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82eae00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82eae00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e40c3b80 con 0x7f82e4151ce0 2024-03-19T02:57:25.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82eae00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f82e4158860 con 0x7f82e4151900 2024-03-19T02:57:25.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82eae00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f82e4072e70 con 0x7f82e40c7da0 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82e3e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82e8e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82e8e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:60522/0 (socket says 172.21.15.92:60522) 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82e3e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:53174/0 (socket says 172.21.15.92:53174) 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82e8e00640 1 -- 172.21.15.92:0/3487040808 learned_addr learned my addr 172.21.15.92:0/3487040808 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.881+0000 7f82dbe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f82e4151ce0 0x7f82e40723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:33892/0 (socket says 172.21.15.92:33892) 2024-03-19T02:57:25.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.882+0000 7f82e8e00640 1 -- 172.21.15.92:0/3487040808 >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 msgr2=0x7f82e4158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:25.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.882+0000 7f82e8e00640 1 --2- 172.21.15.92:0/3487040808 >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.882+0000 7f82e8e00640 1 -- 172.21.15.92:0/3487040808 >> v1:172.21.15.159:6789/0 conn(0x7f82e4151ce0 legacy=0x7f82e40723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:25.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.882+0000 7f82e8e00640 1 -- 172.21.15.92:0/3487040808 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e4073150 con 0x7f82e40c7da0 2024-03-19T02:57:25.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.882+0000 7f82e3e00640 1 --2- 172.21.15.92:0/3487040808 >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:25.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.883+0000 7f82e8e00640 1 --2- 172.21.15.92:0/3487040808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009e50 tx=0x7f82dc02bfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66468757a7d0a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:25.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.883+0000 7f82e3400640 1 -- 172.21.15.92:0/3487040808 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc02cb80 con 0x7f82e40c7da0 2024-03-19T02:57:25.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.883+0000 7f82e3400640 1 -- 172.21.15.92:0/3487040808 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc02cd20 con 0x7f82e40c7da0 2024-03-19T02:57:25.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.883+0000 7f82e3400640 1 -- 172.21.15.92:0/3487040808 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc034610 con 0x7f82e40c7da0 2024-03-19T02:57:25.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 -- 172.21.15.92:0/3487040808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 msgr2=0x7f82e40c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 --2- 172.21.15.92:0/3487040808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009e50 tx=0x7f82dc02bfe0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 -- 172.21.15.92:0/3487040808 shutdown_connections 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 --2- 172.21.15.92:0/3487040808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e40c7da0 0x7f82e40c8230 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 --2- 172.21.15.92:0/3487040808 >> v2:172.21.15.159:3300/0 conn(0x7f82e4151900 0x7f82e4158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 -- 172.21.15.92:0/3487040808 >> 172.21.15.92:0/3487040808 conn(0x7f82e40c8d90 msgr2=0x7f82e40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 -- 172.21.15.92:0/3487040808 shutdown_connections 2024-03-19T02:57:25.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.884+0000 7f82eae00640 1 -- 172.21.15.92:0/3487040808 wait complete. 2024-03-19T02:57:25.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.886+0000 7f82eae00640 1 Processor -- start 2024-03-19T02:57:25.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.886+0000 7f82eae00640 1 -- start start 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 0x7f82e414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 0x7f82e41e1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f82e40c9af0 con 0x7f82e4151ce0 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4071eb0 con 0x7f82e40c7da0 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82eae00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f82e40c9c70 con 0x7f82e4151900 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82e8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82e8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45560/0 (socket says 172.21.15.92:45560) 2024-03-19T02:57:25.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82e8e00640 1 -- 172.21.15.92:0/1068120987 learned_addr learned my addr 172.21.15.92:0/1068120987 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82dbe00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 0x7f82e41e1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.887+0000 7f82e3e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 0x7f82e414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 -- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 msgr2=0x7f82e41e1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 0x7f82e41e1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 -- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 msgr2=0x7f82e414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 0x7f82e414bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 -- 172.21.15.92:0/1068120987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e41f3e50 con 0x7f82e4151ce0 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82dbe00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 0x7f82e41e1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e3e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 0x7f82e414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e8e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009f70 tx=0x7f82dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc043020 con 0x7f82e4151ce0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc03e450 con 0x7f82e4151ce0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc009510 con 0x7f82e4151ce0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e41f40f0 con 0x7f82e4151ce0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.888+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e41f46b0 con 0x7f82e4151ce0 2024-03-19T02:57:25.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.890+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e40684c0 con 0x7f82e4151ce0 2024-03-19T02:57:25.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.891+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f82dc046020 con 0x7f82e4151ce0 2024-03-19T02:57:25.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.891+0000 7f82e0c00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 0x7f82c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:25.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.891+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f82dc0ad0a0 con 0x7f82e4151ce0 2024-03-19T02:57:25.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.892+0000 7f82e3e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 0x7f82c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:25.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.892+0000 7f82e3e00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 0x7f82c006e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009560 tx=0x7f82d4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:25.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:25.896+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f82dc04d030 con 0x7f82e4151ce0 2024-03-19T02:57:26.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.133+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e40c4b80 con 0x7f82e4151ce0 2024-03-19T02:57:26.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.135+0000 7f82e0c00640 1 -- 172.21.15.92:0/1068120987 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6009 (secure 0 0 0) 0x7f82dc07b4e0 con 0x7f82e4151ce0 2024-03-19T02:57:26.136 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:26.136 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:22.102045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:26.137 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 12 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 msgr2=0x7f82c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 0x7f82c006e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009560 tx=0x7f82d4009190 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 msgr2=0x7f82e41f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009f70 tx=0x7f82dc004a40 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 shutdown_connections 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f82c006bbd0 0x7f82c006e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82e4151ce0 0x7f82e41f0840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82e4151900 0x7f82e41e1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 --2- 172.21.15.92:0/1068120987 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e40c7da0 0x7f82e414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.137+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 >> 172.21.15.92:0/1068120987 conn(0x7f82e40c8d90 msgr2=0x7f82e4072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.138+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 shutdown_connections 2024-03-19T02:57:26.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:26.138+0000 7f82eae00640 1 -- 172.21.15.92:0/1068120987 wait complete. 2024-03-19T02:57:26.158 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:28.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:28.365 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.364+0000 7fc012000640 1 Processor -- start 2024-03-19T02:57:28.365 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.364+0000 7fc012000640 1 -- start start 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.365+0000 7fc012000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 0x7fc00c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc012000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c15b740 con 0x7fc00c0c60d0 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc012000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c15ba20 con 0x7fc00c072380 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc012000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc00c0c8400 con 0x7fc00c154dc0 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 0x7fc00c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:28.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00be00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc00c072380 0x7fc00c0c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:33898/0 (socket says 172.21.15.92:33898) 2024-03-19T02:57:28.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00be00640 1 -- 172.21.15.92:0/2787979658 learned_addr learned my addr 172.21.15.92:0/2787979658 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:28.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00b400640 1 -- 172.21.15.92:0/2787979658 >> v1:172.21.15.159:6789/0 conn(0x7fc00c072380 legacy=0x7fc00c0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00b400640 1 -- 172.21.15.92:0/2787979658 >> v1:172.21.15.92:6789/0 conn(0x7fc00c0c60d0 legacy=0x7fc00c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.366+0000 7fc00b400640 1 -- 172.21.15.92:0/2787979658 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c0c86e0 con 0x7fc00c154dc0 2024-03-19T02:57:28.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.367+0000 7fc00b400640 1 --2- 172.21.15.92:0/2787979658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 0x7fc00c071190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097d0 tx=0x7fbffc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=740cd0191a32b182 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:28.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.368+0000 7fc00aa00640 1 -- 172.21.15.92:0/2787979658 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc035020 con 0x7fc00c154dc0 2024-03-19T02:57:28.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.368+0000 7fc00aa00640 1 -- 172.21.15.92:0/2787979658 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbffc031800 con 0x7fc00c154dc0 2024-03-19T02:57:28.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.368+0000 7fc00aa00640 1 -- 172.21.15.92:0/2787979658 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc031bd0 con 0x7fc00c154dc0 2024-03-19T02:57:28.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.368+0000 7fc012000640 1 -- 172.21.15.92:0/2787979658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 msgr2=0x7fc00c071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.368+0000 7fc012000640 1 --2- 172.21.15.92:0/2787979658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 0x7fc00c071190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097d0 tx=0x7fbffc02b8a0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.369+0000 7fc012000640 1 -- 172.21.15.92:0/2787979658 shutdown_connections 2024-03-19T02:57:28.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.369+0000 7fc012000640 1 --2- 172.21.15.92:0/2787979658 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c154dc0 0x7fc00c071190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.369+0000 7fc012000640 1 -- 172.21.15.92:0/2787979658 >> 172.21.15.92:0/2787979658 conn(0x7fc00c0c8d90 msgr2=0x7fc00c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:28.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.369+0000 7fc012000640 1 -- 172.21.15.92:0/2787979658 shutdown_connections 2024-03-19T02:57:28.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.369+0000 7fc012000640 1 -- 172.21.15.92:0/2787979658 wait complete. 2024-03-19T02:57:28.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.370+0000 7fc012000640 1 Processor -- start 2024-03-19T02:57:28.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 -- start start 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc00c154dc0 0x7fc00c148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c0c7520 con 0x7fc00c154dc0 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c0c73a0 con 0x7fc00c072380 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.371+0000 7fc012000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc00c0c76a0 con 0x7fc00c0c60d0 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc00b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc00b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:53196/0 (socket says 172.21.15.92:53196) 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:60538/0 (socket says 172.21.15.92:60538) 2024-03-19T02:57:28.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc00b400640 1 -- 172.21.15.92:0/3488831332 learned_addr learned my addr 172.21.15.92:0/3488831332 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:28.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc00be00640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc00c154dc0 0x7fc00c148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 -- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 msgr2=0x7fc00c14fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 -- 172.21.15.92:0/3488831332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc00c154dc0 msgr2=0x7fc00c148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc00c154dc0 0x7fc00c148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 -- 172.21.15.92:0/3488831332 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c1503b0 con 0x7fc00c0c60d0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc00b400640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.372+0000 7fc003e00640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c370 tx=0x7fbff400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.373+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40052d0 con 0x7fc00c0c60d0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.373+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff4005490 con 0x7fc00c0c60d0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.373+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c1f07b0 con 0x7fc00c0c60d0 2024-03-19T02:57:28.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.373+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4007260 con 0x7fc00c0c60d0 2024-03-19T02:57:28.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.373+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c1f0d30 con 0x7fc00c0c60d0 2024-03-19T02:57:28.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.374+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c06f460 con 0x7fc00c0c60d0 2024-03-19T02:57:28.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.375+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbff4007400 con 0x7fc00c0c60d0 2024-03-19T02:57:28.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.375+0000 7fc003400640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 0x7fbfe806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:28.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.376+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7fbff408a090 con 0x7fc00c0c60d0 2024-03-19T02:57:28.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.376+0000 7fc00b400640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 0x7fbfe806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:28.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.376+0000 7fc00b400640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 0x7fbfe806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbffc000c00 tx=0x7fbffc03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:28.381 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.380+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc00c06f460 con 0x7fc00c0c60d0 2024-03-19T02:57:28.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.726+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc00c0684c0 con 0x7fc00c0c60d0 2024-03-19T02:57:28.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.726+0000 7fc003400640 1 -- 172.21.15.92:0/3488831332 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6009 (secure 0 0 0) 0x7fbff4058310 con 0x7fc00c0c60d0 2024-03-19T02:57:28.728 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:28.728 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:22.102045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:57:28.728 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 12 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 msgr2=0x7fbfe806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 0x7fbfe806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbffc000c00 tx=0x7fbffc03c040 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 msgr2=0x7fc00c15a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c370 tx=0x7fbff400c830 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 shutdown_connections 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbfe806bbd0 0x7fbfe806e090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc00c154dc0 0x7fc00c148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc00c0c60d0 0x7fc00c15a710 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 --2- 172.21.15.92:0/3488831332 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc00c072380 0x7fc00c14fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:28.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.729+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 >> 172.21.15.92:0/3488831332 conn(0x7fc00c0c8d90 msgr2=0x7fc00c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:28.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.730+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 shutdown_connections 2024-03-19T02:57:28.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:28.730+0000 7fc012000640 1 -- 172.21.15.92:0/3488831332 wait complete. 2024-03-19T02:57:28.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:57:29.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:57:29.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:57:30.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:30.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.972+0000 7f6c2d800640 1 Processor -- start 2024-03-19T02:57:30.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.972+0000 7f6c2d800640 1 -- start start 2024-03-19T02:57:30.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c2d800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c280c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:30.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c2d800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c280c9a80 con 0x7f6c281593d0 2024-03-19T02:57:30.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c2d800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c280c8650 con 0x7f6c28152bf0 2024-03-19T02:57:30.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c2d800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6c280c6c90 con 0x7f6c280c6470 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c27400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6c281593d0 0x7f6c280c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:37064/0 (socket says 172.21.15.92:37064) 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c27400640 1 -- 172.21.15.92:0/1126256686 learned_addr learned my addr 172.21.15.92:0/1126256686 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c26000640 1 --2- 172.21.15.92:0/1126256686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c280c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.974+0000 7f6c25600640 1 -- 172.21.15.92:0/1126256686 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706616531 0 0) 0x7f6c280c9a80 con 0x7f6c281593d0 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.975+0000 7f6c26000640 1 -- 172.21.15.92:0/1126256686 >> v1:172.21.15.159:6789/0 conn(0x7f6c28152bf0 legacy=0x7f6c2806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.975+0000 7f6c26000640 1 -- 172.21.15.92:0/1126256686 >> v1:172.21.15.92:6789/0 conn(0x7f6c281593d0 legacy=0x7f6c280c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:30.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.975+0000 7f6c26000640 1 -- 172.21.15.92:0/1126256686 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c280c8930 con 0x7f6c280c6470 2024-03-19T02:57:30.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c26000640 1 --2- 172.21.15.92:0/1126256686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c280c3f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009750 tx=0x7f6c1002b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a5aba534bc13ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:30.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c25600640 1 -- 172.21.15.92:0/1126256686 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10037070 con 0x7f6c280c6470 2024-03-19T02:57:30.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c25600640 1 -- 172.21.15.92:0/1126256686 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c10030710 con 0x7f6c280c6470 2024-03-19T02:57:30.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c25600640 1 -- 172.21.15.92:0/1126256686 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10030ae0 con 0x7f6c280c6470 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c2d800640 1 -- 172.21.15.92:0/1126256686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 msgr2=0x7f6c280c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.976+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1126256686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c280c3f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009750 tx=0x7f6c1002b680 comp rx=0 tx=0).stop 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.977+0000 7f6c2d800640 1 -- 172.21.15.92:0/1126256686 shutdown_connections 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.977+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1126256686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c280c3f80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.977+0000 7f6c2d800640 1 -- 172.21.15.92:0/1126256686 >> 172.21.15.92:0/1126256686 conn(0x7f6c280c8d90 msgr2=0x7f6c280c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.977+0000 7f6c2d800640 1 -- 172.21.15.92:0/1126256686 shutdown_connections 2024-03-19T02:57:30.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.977+0000 7f6c2d800640 1 -- 172.21.15.92:0/1126256686 wait complete. 2024-03-19T02:57:30.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.978+0000 7f6c2d800640 1 Processor -- start 2024-03-19T02:57:30.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.978+0000 7f6c2d800640 1 -- start start 2024-03-19T02:57:30.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c28148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:30.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:30.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c281593d0 0x7f6c28142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:30.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2806e4a0 con 0x7f6c28152bf0 2024-03-19T02:57:30.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6c28159910 con 0x7f6c281593d0 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c2d800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2806ea60 con 0x7f6c280c6470 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.979+0000 7f6c26000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c28148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45584/0 (socket says 172.21.15.92:45584) 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 -- 172.21.15.92:0/1166700264 learned_addr learned my addr 172.21.15.92:0/1166700264 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c27400640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c281593d0 0x7f6c28142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 -- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 msgr2=0x7f6c28148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c28148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 -- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c281593d0 msgr2=0x7f6c28142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c281593d0 0x7f6c28142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 -- 172.21.15.92:0/1166700264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c281f1b50 con 0x7f6c28152bf0 2024-03-19T02:57:30.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26a00640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c28148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:30.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.980+0000 7f6c26000640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009870 tx=0x7f6c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:30.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.981+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10037040 con 0x7f6c28152bf0 2024-03-19T02:57:30.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.981+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c100329c0 con 0x7f6c28152bf0 2024-03-19T02:57:30.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.981+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c281f1e30 con 0x7f6c28152bf0 2024-03-19T02:57:30.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.981+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10032d90 con 0x7f6c28152bf0 2024-03-19T02:57:30.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.981+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c281f23f0 con 0x7f6c28152bf0 2024-03-19T02:57:30.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.982+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c280684c0 con 0x7f6c28152bf0 2024-03-19T02:57:30.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.983+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6c10004030 con 0x7f6c28152bf0 2024-03-19T02:57:30.985 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.984+0000 7f6c1ea00640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 0x7f6c0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:30.985 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.984+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f6c100ac930 con 0x7f6c28152bf0 2024-03-19T02:57:30.985 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.984+0000 7f6c26a00640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 0x7f6c0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:30.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.985+0000 7f6c26a00640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 0x7f6c0406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6c18000f60 tx=0x7f6c18009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:30.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:30.989+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6c10042020 con 0x7f6c28152bf0 2024-03-19T02:57:31.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.221+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c280c3b50 con 0x7f6c28152bf0 2024-03-19T02:57:31.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.222+0000 7f6c1ea00640 1 -- 172.21.15.92:0/1166700264 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6010 (secure 0 0 0) 0x7f6c1007abb0 con 0x7f6c28152bf0 2024-03-19T02:57:31.224 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:31.224 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:57:31.225 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 13 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 msgr2=0x7f6c0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 0x7f6c0406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6c18000f60 tx=0x7f6c18009040 comp rx=0 tx=0).stop 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 msgr2=0x7f6c28150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009870 tx=0x7f6c100046c0 comp rx=0 tx=0).stop 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 shutdown_connections 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6c0406bbd0 0x7f6c0406e090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6c281593d0 0x7f6c28142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c28152bf0 0x7f6c28150e90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:31.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 --2- 172.21.15.92:0/1166700264 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6c280c6470 0x7f6c28148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:31.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.225+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 >> 172.21.15.92:0/1166700264 conn(0x7f6c280c8d90 msgr2=0x7f6c280c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:31.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.226+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 shutdown_connections 2024-03-19T02:57:31.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:31.226+0000 7f6c2d800640 1 -- 172.21.15.92:0/1166700264 wait complete. 2024-03-19T02:57:31.244 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4264 2024-03-19T02:57:31.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2024-03-19T02:57:34.246 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-03-19T02:57:34.246 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-19T02:57:34.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-19T02:57:34.249 INFO:tasks.ceph.mds.d:Started 2024-03-19T02:57:34.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:34.332 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-19T02:57:34.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.468+0000 7f999b200640 1 Processor -- start 2024-03-19T02:57:34.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.469+0000 7f999b200640 1 -- start start 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.470+0000 7f999b200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f999c0c5480 0x7f999c072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.470+0000 7f999b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 0x7f999c151610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.470+0000 7f999b200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f999c072140 con 0x7f999c0c5860 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.470+0000 7f999b200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f999c159520 con 0x7f999c15a0c0 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.470+0000 7f999b200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f999c0c4ef0 con 0x7f999c0c5480 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f999c0c5480 0x7f999c072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f999c0c5480 0x7f999c072390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:53202/0 (socket says 172.21.15.92:53202) 2024-03-19T02:57:34.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 -- 172.21.15.92:0/4027344690 learned_addr learned my addr 172.21.15.92:0/4027344690 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f999a800640 1 --2- 172.21.15.92:0/4027344690 >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 0x7f999c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 -- 172.21.15.92:0/4027344690 >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 msgr2=0x7f999c151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 --2- 172.21.15.92:0/4027344690 >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 0x7f999c151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 -- 172.21.15.92:0/4027344690 >> v1:172.21.15.159:6789/0 conn(0x7f999c0c5860 legacy=0x7f999c1599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f9999e00640 1 -- 172.21.15.92:0/4027344690 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c0728d0 con 0x7f999c0c5480 2024-03-19T02:57:34.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.471+0000 7f999a800640 1 --2- 172.21.15.92:0/4027344690 >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 0x7f999c151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:57:34.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.472+0000 7f9999e00640 1 --2- 172.21.15.92:0/4027344690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c072390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f999c070890 tx=0x7f998c02b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616f6080a89552d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.472+0000 7f9998a00640 1 -- 172.21.15.92:0/4027344690 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c035020 con 0x7f999c0c5480 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.472+0000 7f9998a00640 1 -- 172.21.15.92:0/4027344690 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f998c031880 con 0x7f999c0c5480 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.472+0000 7f9998a00640 1 -- 172.21.15.92:0/4027344690 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c031c50 con 0x7f999c0c5480 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 -- 172.21.15.92:0/4027344690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 msgr2=0x7f999c072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 --2- 172.21.15.92:0/4027344690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c072390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f999c070890 tx=0x7f998c02b920 comp rx=0 tx=0).stop 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 -- 172.21.15.92:0/4027344690 shutdown_connections 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 --2- 172.21.15.92:0/4027344690 >> v2:172.21.15.92:3300/0 conn(0x7f999c15a0c0 0x7f999c151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 --2- 172.21.15.92:0/4027344690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c072390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 -- 172.21.15.92:0/4027344690 >> 172.21.15.92:0/4027344690 conn(0x7f999c0cc1f0 msgr2=0x7f999c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 -- 172.21.15.92:0/4027344690 shutdown_connections 2024-03-19T02:57:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.473+0000 7f999b200640 1 -- 172.21.15.92:0/4027344690 wait complete. 2024-03-19T02:57:34.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.475+0000 7f999b200640 1 Processor -- start 2024-03-19T02:57:34.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.475+0000 7f999b200640 1 -- start start 2024-03-19T02:57:34.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c14f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 0x7f999c153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f999c158ed0 con 0x7f999c0c5860 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f999c071e60 con 0x7f999c0c5480 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999b200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f999c1591b0 con 0x7f999c15a0c0 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f9999400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f999a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 0x7f999c153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f9999400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45608/0 (socket says 172.21.15.92:45608) 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f9999400640 1 -- 172.21.15.92:0/3220005065 learned_addr learned my addr 172.21.15.92:0/3220005065 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:34.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.476+0000 7f9999e00640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c14f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 -- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 msgr2=0x7f999c14f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c14f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 -- 172.21.15.92:0/3220005065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 msgr2=0x7f999c155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 -- 172.21.15.92:0/3220005065 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c1557e0 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f9999400640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f9999e00640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c14f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999a800640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 0x7f999c153850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f999400ccc0 tx=0x7f999400a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999400f680 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999400f840 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999c1ec810 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999400fc30 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.477+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f999c1ecdd0 con 0x7f999c15a0c0 2024-03-19T02:57:34.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.478+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f999c068de0 con 0x7f999c15a0c0 2024-03-19T02:57:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.479+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f99940213f0 con 0x7f999c15a0c0 2024-03-19T02:57:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.480+0000 7f9992000640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 0x7f997806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.480+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7f999408b730 con 0x7f999c15a0c0 2024-03-19T02:57:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.480+0000 7f9999e00640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 0x7f997806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.481+0000 7f9999e00640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 0x7f997806e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f998c036000 tx=0x7f998c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:34.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.484+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f999c068de0 con 0x7f999c15a0c0 2024-03-19T02:57:34.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.720+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f999c04f580 con 0x7f999c15a0c0 2024-03-19T02:57:34.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.721+0000 7f9992000640 1 -- 172.21.15.92:0/3220005065 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6010 (secure 0 0 0) 0x7f9994014030 con 0x7f999c15a0c0 2024-03-19T02:57:34.721 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:34.722 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:57:34.722 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 13 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.723+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 msgr2=0x7f997806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.723+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 0x7f997806e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f998c036000 tx=0x7f998c03c040 comp rx=0 tx=0).stop 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 msgr2=0x7f999c153850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 0x7f999c153850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f999400ccc0 tx=0x7f999400a650 comp rx=0 tx=0).stop 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 shutdown_connections 2024-03-19T02:57:34.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f997806bbd0 0x7f997806e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f999c15a0c0 0x7f999c153850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f999c0c5860 0x7f999c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 --2- 172.21.15.92:0/3220005065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f999c0c5480 0x7f999c14f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 >> 172.21.15.92:0/3220005065 conn(0x7f999c0cc1f0 msgr2=0x7f999c06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 shutdown_connections 2024-03-19T02:57:34.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:34.724+0000 7f999b200640 1 -- 172.21.15.92:0/3220005065 wait complete. 2024-03-19T02:57:34.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-19T02:57:36.160 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/35 file after the snapshot 2024-03-19T02:57:36.163 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/36 file after the snapshot 2024-03-19T02:57:36.164 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/37 file after the snapshot 2024-03-19T02:57:36.166 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/38 file after the snapshot 2024-03-19T02:57:36.167 INFO:tasks.workunit.client.0.smithi092.stdout: create 187/39 file after the snapshot 2024-03-19T02:57:36.169 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/.snap/snap-subdir-test 2024-03-19T02:57:36.211 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/30 file after the snapshot 2024-03-19T02:57:36.213 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/31 file after the snapshot 2024-03-19T02:57:36.215 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/32 file after the snapshot 2024-03-19T02:57:36.216 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/33 file after the snapshot 2024-03-19T02:57:36.217 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/34 file after the snapshot 2024-03-19T02:57:36.219 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/35 file after the snapshot 2024-03-19T02:57:36.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/36 file after the snapshot 2024-03-19T02:57:36.222 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/37 file after the snapshot 2024-03-19T02:57:36.224 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/38 file after the snapshot 2024-03-19T02:57:36.226 INFO:tasks.workunit.client.0.smithi092.stdout: create 188/39 file after the snapshot 2024-03-19T02:57:36.228 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/.snap/snap-subdir-test 2024-03-19T02:57:36.270 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/30 file after the snapshot 2024-03-19T02:57:36.272 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/31 file after the snapshot 2024-03-19T02:57:36.274 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/32 file after the snapshot 2024-03-19T02:57:36.276 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/33 file after the snapshot 2024-03-19T02:57:36.278 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/34 file after the snapshot 2024-03-19T02:57:36.280 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/35 file after the snapshot 2024-03-19T02:57:36.281 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/36 file after the snapshot 2024-03-19T02:57:36.283 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/37 file after the snapshot 2024-03-19T02:57:36.285 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/38 file after the snapshot 2024-03-19T02:57:36.287 INFO:tasks.workunit.client.0.smithi092.stdout: create 189/39 file after the snapshot 2024-03-19T02:57:36.289 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/.snap/snap-subdir-test 2024-03-19T02:57:36.329 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/30 file after the snapshot 2024-03-19T02:57:36.331 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/31 file after the snapshot 2024-03-19T02:57:36.333 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/32 file after the snapshot 2024-03-19T02:57:36.334 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/33 file after the snapshot 2024-03-19T02:57:36.336 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/34 file after the snapshot 2024-03-19T02:57:36.338 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/35 file after the snapshot 2024-03-19T02:57:36.340 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/36 file after the snapshot 2024-03-19T02:57:36.342 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/37 file after the snapshot 2024-03-19T02:57:36.344 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/38 file after the snapshot 2024-03-19T02:57:36.345 INFO:tasks.workunit.client.0.smithi092.stdout: create 190/39 file after the snapshot 2024-03-19T02:57:36.347 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/.snap/snap-subdir-test 2024-03-19T02:57:36.390 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/30 file after the snapshot 2024-03-19T02:57:36.392 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/31 file after the snapshot 2024-03-19T02:57:36.394 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/32 file after the snapshot 2024-03-19T02:57:36.396 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/33 file after the snapshot 2024-03-19T02:57:36.397 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/34 file after the snapshot 2024-03-19T02:57:36.399 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/35 file after the snapshot 2024-03-19T02:57:36.401 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/36 file after the snapshot 2024-03-19T02:57:36.403 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/37 file after the snapshot 2024-03-19T02:57:36.404 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/38 file after the snapshot 2024-03-19T02:57:36.405 INFO:tasks.workunit.client.0.smithi092.stdout: create 191/39 file after the snapshot 2024-03-19T02:57:36.407 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/.snap/snap-subdir-test 2024-03-19T02:57:36.447 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/30 file after the snapshot 2024-03-19T02:57:36.450 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/31 file after the snapshot 2024-03-19T02:57:36.451 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/32 file after the snapshot 2024-03-19T02:57:36.454 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/33 file after the snapshot 2024-03-19T02:57:36.455 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/34 file after the snapshot 2024-03-19T02:57:36.457 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/35 file after the snapshot 2024-03-19T02:57:36.459 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/36 file after the snapshot 2024-03-19T02:57:36.461 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/37 file after the snapshot 2024-03-19T02:57:36.463 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/38 file after the snapshot 2024-03-19T02:57:36.465 INFO:tasks.workunit.client.0.smithi092.stdout: create 192/39 file after the snapshot 2024-03-19T02:57:36.467 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/.snap/snap-subdir-test 2024-03-19T02:57:36.509 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/30 file after the snapshot 2024-03-19T02:57:36.510 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/31 file after the snapshot 2024-03-19T02:57:36.511 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/32 file after the snapshot 2024-03-19T02:57:36.513 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/33 file after the snapshot 2024-03-19T02:57:36.514 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/34 file after the snapshot 2024-03-19T02:57:36.516 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/35 file after the snapshot 2024-03-19T02:57:36.517 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/36 file after the snapshot 2024-03-19T02:57:36.518 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/37 file after the snapshot 2024-03-19T02:57:36.520 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/38 file after the snapshot 2024-03-19T02:57:36.521 INFO:tasks.workunit.client.0.smithi092.stdout: create 193/39 file after the snapshot 2024-03-19T02:57:36.522 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/.snap/snap-subdir-test 2024-03-19T02:57:36.564 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/30 file after the snapshot 2024-03-19T02:57:36.565 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/31 file after the snapshot 2024-03-19T02:57:36.567 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/32 file after the snapshot 2024-03-19T02:57:36.568 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/33 file after the snapshot 2024-03-19T02:57:36.570 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/34 file after the snapshot 2024-03-19T02:57:36.572 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/35 file after the snapshot 2024-03-19T02:57:36.573 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/36 file after the snapshot 2024-03-19T02:57:36.575 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/37 file after the snapshot 2024-03-19T02:57:36.576 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/38 file after the snapshot 2024-03-19T02:57:36.578 INFO:tasks.workunit.client.0.smithi092.stdout: create 194/39 file after the snapshot 2024-03-19T02:57:36.579 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/.snap/snap-subdir-test 2024-03-19T02:57:36.621 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/30 file after the snapshot 2024-03-19T02:57:36.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/31 file after the snapshot 2024-03-19T02:57:36.625 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/32 file after the snapshot 2024-03-19T02:57:36.627 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/33 file after the snapshot 2024-03-19T02:57:36.629 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/34 file after the snapshot 2024-03-19T02:57:36.631 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/35 file after the snapshot 2024-03-19T02:57:36.633 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/36 file after the snapshot 2024-03-19T02:57:36.634 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/37 file after the snapshot 2024-03-19T02:57:36.636 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/38 file after the snapshot 2024-03-19T02:57:36.638 INFO:tasks.workunit.client.0.smithi092.stdout: create 195/39 file after the snapshot 2024-03-19T02:57:36.640 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/.snap/snap-subdir-test 2024-03-19T02:57:36.685 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/30 file after the snapshot 2024-03-19T02:57:36.686 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/31 file after the snapshot 2024-03-19T02:57:36.687 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/32 file after the snapshot 2024-03-19T02:57:36.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/33 file after the snapshot 2024-03-19T02:57:36.690 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/34 file after the snapshot 2024-03-19T02:57:36.692 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/35 file after the snapshot 2024-03-19T02:57:36.693 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/36 file after the snapshot 2024-03-19T02:57:36.694 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/37 file after the snapshot 2024-03-19T02:57:36.696 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/38 file after the snapshot 2024-03-19T02:57:36.697 INFO:tasks.workunit.client.0.smithi092.stdout: create 196/39 file after the snapshot 2024-03-19T02:57:36.699 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/.snap/snap-subdir-test 2024-03-19T02:57:36.742 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/30 file after the snapshot 2024-03-19T02:57:36.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:36.744 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/31 file after the snapshot 2024-03-19T02:57:36.745 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/32 file after the snapshot 2024-03-19T02:57:36.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/33 file after the snapshot 2024-03-19T02:57:36.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/34 file after the snapshot 2024-03-19T02:57:36.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/35 file after the snapshot 2024-03-19T02:57:36.752 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/36 file after the snapshot 2024-03-19T02:57:36.752 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/37 file after the snapshot 2024-03-19T02:57:36.753 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/38 file after the snapshot 2024-03-19T02:57:36.755 INFO:tasks.workunit.client.0.smithi092.stdout: create 197/39 file after the snapshot 2024-03-19T02:57:36.756 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/.snap/snap-subdir-test 2024-03-19T02:57:36.786 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/30 file after the snapshot 2024-03-19T02:57:36.787 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/31 file after the snapshot 2024-03-19T02:57:36.788 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/32 file after the snapshot 2024-03-19T02:57:36.789 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/33 file after the snapshot 2024-03-19T02:57:36.790 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/34 file after the snapshot 2024-03-19T02:57:36.791 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/35 file after the snapshot 2024-03-19T02:57:36.792 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/36 file after the snapshot 2024-03-19T02:57:36.793 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/37 file after the snapshot 2024-03-19T02:57:36.794 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/38 file after the snapshot 2024-03-19T02:57:36.795 INFO:tasks.workunit.client.0.smithi092.stdout: create 198/39 file after the snapshot 2024-03-19T02:57:36.796 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/.snap/snap-subdir-test 2024-03-19T02:57:36.820 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/30 file after the snapshot 2024-03-19T02:57:36.821 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/31 file after the snapshot 2024-03-19T02:57:36.823 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/32 file after the snapshot 2024-03-19T02:57:36.824 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/33 file after the snapshot 2024-03-19T02:57:36.825 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/34 file after the snapshot 2024-03-19T02:57:36.826 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/35 file after the snapshot 2024-03-19T02:57:36.827 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/36 file after the snapshot 2024-03-19T02:57:36.830 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/37 file after the snapshot 2024-03-19T02:57:36.831 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/38 file after the snapshot 2024-03-19T02:57:36.832 INFO:tasks.workunit.client.0.smithi092.stdout: create 199/39 file after the snapshot 2024-03-19T02:57:36.833 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/.snap/snap-subdir-test 2024-03-19T02:57:36.857 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/30 file after the snapshot 2024-03-19T02:57:36.861 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/31 file after the snapshot 2024-03-19T02:57:36.862 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/32 file after the snapshot 2024-03-19T02:57:36.864 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/33 file after the snapshot 2024-03-19T02:57:36.865 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/34 file after the snapshot 2024-03-19T02:57:36.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/35 file after the snapshot 2024-03-19T02:57:36.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/36 file after the snapshot 2024-03-19T02:57:36.870 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/37 file after the snapshot 2024-03-19T02:57:36.871 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/38 file after the snapshot 2024-03-19T02:57:36.873 INFO:tasks.workunit.client.0.smithi092.stdout: create 200/39 file after the snapshot 2024-03-19T02:57:36.875 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/.snap/snap-subdir-test 2024-03-19T02:57:36.904 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/30 file after the snapshot 2024-03-19T02:57:36.906 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/31 file after the snapshot 2024-03-19T02:57:36.908 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/32 file after the snapshot 2024-03-19T02:57:36.909 INFO:tasks.ceph.mds.f.smithi159.stderr:2024-03-19T02:57:36.908+0000 7fb6b5c95640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T02:57:36.909 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/33 file after the snapshot 2024-03-19T02:57:36.911 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/34 file after the snapshot 2024-03-19T02:57:36.913 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/35 file after the snapshot 2024-03-19T02:57:36.914 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/36 file after the snapshot 2024-03-19T02:57:36.916 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/37 file after the snapshot 2024-03-19T02:57:36.918 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/38 file after the snapshot 2024-03-19T02:57:36.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 201/39 file after the snapshot 2024-03-19T02:57:36.921 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/.snap/snap-subdir-test 2024-03-19T02:57:36.951 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/30 file after the snapshot 2024-03-19T02:57:36.953 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/31 file after the snapshot 2024-03-19T02:57:36.955 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/32 file after the snapshot 2024-03-19T02:57:36.957 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/33 file after the snapshot 2024-03-19T02:57:36.958 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/34 file after the snapshot 2024-03-19T02:57:36.960 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/35 file after the snapshot 2024-03-19T02:57:36.962 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/36 file after the snapshot 2024-03-19T02:57:36.963 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/37 file after the snapshot 2024-03-19T02:57:36.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.963+0000 7efd03200640 1 Processor -- start 2024-03-19T02:57:36.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.963+0000 7efd03200640 1 -- start start 2024-03-19T02:57:36.964 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/38 file after the snapshot 2024-03-19T02:57:36.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd03200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efd040c5230 0x7efd040c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd03200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd040c9730 con 0x7efd04156060 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd03200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0406fc60 con 0x7efd04156440 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd03200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7efd0415c7e0 con 0x7efd040c5230 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd01e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efd04156060 0x7efd0415c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:52786/0 (socket says 172.21.15.92:52786) 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd01e00640 1 -- 172.21.15.92:0/1978247949 learned_addr learned my addr 172.21.15.92:0/1978247949 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:57:36.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.965+0000 7efd02800640 1 --2- 172.21.15.92:0/1978247949 >> v2:172.21.15.159:3300/0 conn(0x7efd040c5230 0x7efd040c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.966 INFO:tasks.workunit.client.0.smithi092.stdout: create 202/39 file after the snapshot 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955989600 0 0) 0x7efd040c9730 con 0x7efd04156060 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efce8003520 con 0x7efd04156060 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878828687 0 0) 0x7efd0406fc60 con 0x7efd04156440 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd040c9730 con 0x7efd04156440 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd02800640 1 -- 172.21.15.92:0/1978247949 >> v1:172.21.15.159:6789/0 conn(0x7efd04156440 legacy=0x7efd040ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd02800640 1 -- 172.21.15.92:0/1978247949 >> v1:172.21.15.92:6789/0 conn(0x7efd04156060 legacy=0x7efd0415c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.966+0000 7efd02800640 1 -- 172.21.15.92:0/1978247949 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0406ff40 con 0x7efd040c5230 2024-03-19T02:57:36.968 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/.snap/snap-subdir-test 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.967+0000 7efd02800640 1 --2- 172.21.15.92:0/1978247949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd040c5230 0x7efd040c8dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009d90 tx=0x7efcfc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8220306c1617943d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.967+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc039070 con 0x7efd040c5230 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.967+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcfc02cd00 con 0x7efd040c5230 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.967+0000 7efd00a00640 1 -- 172.21.15.92:0/1978247949 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc034640 con 0x7efd040c5230 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 -- 172.21.15.92:0/1978247949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd040c5230 msgr2=0x7efd040c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 --2- 172.21.15.92:0/1978247949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd040c5230 0x7efd040c8dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009d90 tx=0x7efcfc02bf50 comp rx=0 tx=0).stop 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 -- 172.21.15.92:0/1978247949 shutdown_connections 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 --2- 172.21.15.92:0/1978247949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd040c5230 0x7efd040c8dc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 -- 172.21.15.92:0/1978247949 >> 172.21.15.92:0/1978247949 conn(0x7efd0406b660 msgr2=0x7efd04066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 -- 172.21.15.92:0/1978247949 shutdown_connections 2024-03-19T02:57:36.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.968+0000 7efd03200640 1 -- 172.21.15.92:0/1978247949 wait complete. 2024-03-19T02:57:36.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.970+0000 7efd03200640 1 Processor -- start 2024-03-19T02:57:36.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.970+0000 7efd03200640 1 -- start start 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd04156060 0x7efd0415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 0x7efd04145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efd040cecb0 con 0x7efd04156060 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efd040ce790 con 0x7efd04144d00 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd03200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efd040cd210 con 0x7efd040c5230 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd01e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd02800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 0x7efd04145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd01e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46808/0 (socket says 172.21.15.92:46808) 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd01e00640 1 -- 172.21.15.92:0/1232977986 learned_addr learned my addr 172.21.15.92:0/1232977986 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:57:36.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.971+0000 7efd01400640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd04156060 0x7efd0415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 -- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 msgr2=0x7efd04145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 0x7efd04145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 -- 172.21.15.92:0/1232977986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd04156060 msgr2=0x7efd0415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd04156060 0x7efd0415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 -- 172.21.15.92:0/1232977986 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd04148bf0 con 0x7efd040c5230 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd02800640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 0x7efd04145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efd01e00640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c490 tx=0x7efcf400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4017070 con 0x7efd040c5230 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.972+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf4005d00 con 0x7efd040c5230 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.973+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd041f1380 con 0x7efd040c5230 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.973+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4007310 con 0x7efd040c5230 2024-03-19T02:57:36.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.973+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd040cd740 con 0x7efd040c5230 2024-03-19T02:57:36.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.974+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0406bc50 con 0x7efd040c5230 2024-03-19T02:57:36.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.975+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efcf401a070 con 0x7efd040c5230 2024-03-19T02:57:36.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.976+0000 7efcfa000640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 0x7efce406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.976+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5940+0+0 (secure 0 0 0) 0x7efcf4089fc0 con 0x7efd040c5230 2024-03-19T02:57:36.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.976+0000 7efd01400640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 0x7efce406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.977+0000 7efd01400640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 0x7efce406e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efcf0007fd0 tx=0x7efcf0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:36.982+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efcf4058350 con 0x7efd040c5230 2024-03-19T02:57:36.994 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/30 file after the snapshot 2024-03-19T02:57:36.996 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/31 file after the snapshot 2024-03-19T02:57:36.998 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/32 file after the snapshot 2024-03-19T02:57:36.999 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/33 file after the snapshot 2024-03-19T02:57:37.001 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/34 file after the snapshot 2024-03-19T02:57:37.002 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/35 file after the snapshot 2024-03-19T02:57:37.004 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/36 file after the snapshot 2024-03-19T02:57:37.006 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/37 file after the snapshot 2024-03-19T02:57:37.007 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/38 file after the snapshot 2024-03-19T02:57:37.009 INFO:tasks.workunit.client.0.smithi092.stdout: create 203/39 file after the snapshot 2024-03-19T02:57:37.011 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/.snap/snap-subdir-test 2024-03-19T02:57:37.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/30 file after the snapshot 2024-03-19T02:57:37.041 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/31 file after the snapshot 2024-03-19T02:57:37.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.223+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd040c8cc0 con 0x7efd040c5230 2024-03-19T02:57:37.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.224+0000 7efcfa000640 1 -- 172.21.15.92:0/1232977986 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7efd040c8cc0 con 0x7efd040c5230 2024-03-19T02:57:37.225 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:57:37.225 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:57:37.225 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 msgr2=0x7efce406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 0x7efce406e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efcf0007fd0 tx=0x7efcf0009040 comp rx=0 tx=0).stop 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 msgr2=0x7efd0415afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c490 tx=0x7efcf400c950 comp rx=0 tx=0).stop 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 shutdown_connections 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efce406bc20 0x7efce406e0e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd04144d00 0x7efd04145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd04156060 0x7efd0415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 --2- 172.21.15.92:0/1232977986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd040c5230 0x7efd0415afe0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 >> 172.21.15.92:0/1232977986 conn(0x7efd0406b660 msgr2=0x7efd04153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 shutdown_connections 2024-03-19T02:57:37.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:57:37.227+0000 7efd03200640 1 -- 172.21.15.92:0/1232977986 wait complete. 2024-03-19T02:57:37.246 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-03-19T02:57:37.246 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2024-03-19T02:57:37.304 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/32 file after the snapshot 2024-03-19T02:57:37.305 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/33 file after the snapshot 2024-03-19T02:57:37.307 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/34 file after the snapshot 2024-03-19T02:57:37.309 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/35 file after the snapshot 2024-03-19T02:57:37.312 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/36 file after the snapshot 2024-03-19T02:57:37.314 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/37 file after the snapshot 2024-03-19T02:57:37.315 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/38 file after the snapshot 2024-03-19T02:57:37.317 INFO:tasks.workunit.client.0.smithi092.stdout: create 204/39 file after the snapshot 2024-03-19T02:57:37.319 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/.snap/snap-subdir-test 2024-03-19T02:57:37.354 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/30 file after the snapshot 2024-03-19T02:57:37.356 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/31 file after the snapshot 2024-03-19T02:57:37.358 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/32 file after the snapshot 2024-03-19T02:57:37.360 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/33 file after the snapshot 2024-03-19T02:57:37.361 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/34 file after the snapshot 2024-03-19T02:57:37.363 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/35 file after the snapshot 2024-03-19T02:57:37.365 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/36 file after the snapshot 2024-03-19T02:57:37.367 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/37 file after the snapshot 2024-03-19T02:57:37.369 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/38 file after the snapshot 2024-03-19T02:57:37.371 INFO:tasks.workunit.client.0.smithi092.stdout: create 205/39 file after the snapshot 2024-03-19T02:57:37.373 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/.snap/snap-subdir-test 2024-03-19T02:57:37.406 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/30 file after the snapshot 2024-03-19T02:57:37.408 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/31 file after the snapshot 2024-03-19T02:57:37.410 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/32 file after the snapshot 2024-03-19T02:57:37.412 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/33 file after the snapshot 2024-03-19T02:57:37.413 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/34 file after the snapshot 2024-03-19T02:57:37.415 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/35 file after the snapshot 2024-03-19T02:57:37.417 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/36 file after the snapshot 2024-03-19T02:57:37.419 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/37 file after the snapshot 2024-03-19T02:57:37.421 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/38 file after the snapshot 2024-03-19T02:57:37.426 INFO:tasks.workunit.client.0.smithi092.stdout: create 206/39 file after the snapshot 2024-03-19T02:57:37.426 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/.snap/snap-subdir-test 2024-03-19T02:57:37.457 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/30 file after the snapshot 2024-03-19T02:57:37.459 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/31 file after the snapshot 2024-03-19T02:57:37.461 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/32 file after the snapshot 2024-03-19T02:57:37.463 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/33 file after the snapshot 2024-03-19T02:57:37.465 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/34 file after the snapshot 2024-03-19T02:57:37.466 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/35 file after the snapshot 2024-03-19T02:57:37.468 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/36 file after the snapshot 2024-03-19T02:57:37.470 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/37 file after the snapshot 2024-03-19T02:57:37.472 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/38 file after the snapshot 2024-03-19T02:57:37.474 INFO:tasks.workunit.client.0.smithi092.stdout: create 207/39 file after the snapshot 2024-03-19T02:57:37.476 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/.snap/snap-subdir-test 2024-03-19T02:57:37.507 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/30 file after the snapshot 2024-03-19T02:57:37.509 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/31 file after the snapshot 2024-03-19T02:57:37.511 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/32 file after the snapshot 2024-03-19T02:57:37.513 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/33 file after the snapshot 2024-03-19T02:57:37.515 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/34 file after the snapshot 2024-03-19T02:57:37.516 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/35 file after the snapshot 2024-03-19T02:57:37.518 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/36 file after the snapshot 2024-03-19T02:57:37.520 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/37 file after the snapshot 2024-03-19T02:57:37.522 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/38 file after the snapshot 2024-03-19T02:57:37.524 INFO:tasks.workunit.client.0.smithi092.stdout: create 208/39 file after the snapshot 2024-03-19T02:57:37.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/.snap/snap-subdir-test 2024-03-19T02:57:37.556 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/30 file after the snapshot 2024-03-19T02:57:37.558 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/31 file after the snapshot 2024-03-19T02:57:37.560 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/32 file after the snapshot 2024-03-19T02:57:37.562 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/33 file after the snapshot 2024-03-19T02:57:37.564 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/34 file after the snapshot 2024-03-19T02:57:37.566 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/35 file after the snapshot 2024-03-19T02:57:37.567 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/36 file after the snapshot 2024-03-19T02:57:37.569 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/37 file after the snapshot 2024-03-19T02:57:37.571 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/38 file after the snapshot 2024-03-19T02:57:37.573 INFO:tasks.workunit.client.0.smithi092.stdout: create 209/39 file after the snapshot 2024-03-19T02:57:37.575 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/.snap/snap-subdir-test 2024-03-19T02:57:37.605 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/30 file after the snapshot 2024-03-19T02:57:37.607 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/31 file after the snapshot 2024-03-19T02:57:37.608 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/32 file after the snapshot 2024-03-19T02:57:37.610 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/33 file after the snapshot 2024-03-19T02:57:37.612 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/34 file after the snapshot 2024-03-19T02:57:37.614 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/35 file after the snapshot 2024-03-19T02:57:37.615 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/36 file after the snapshot 2024-03-19T02:57:37.617 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/37 file after the snapshot 2024-03-19T02:57:37.619 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/38 file after the snapshot 2024-03-19T02:57:37.620 INFO:tasks.workunit.client.0.smithi092.stdout: create 210/39 file after the snapshot 2024-03-19T02:57:37.621 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/.snap/snap-subdir-test 2024-03-19T02:57:37.652 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/30 file after the snapshot 2024-03-19T02:57:37.653 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/31 file after the snapshot 2024-03-19T02:57:37.654 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/32 file after the snapshot 2024-03-19T02:57:37.656 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/33 file after the snapshot 2024-03-19T02:57:37.657 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/34 file after the snapshot 2024-03-19T02:57:37.658 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/35 file after the snapshot 2024-03-19T02:57:37.660 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/36 file after the snapshot 2024-03-19T02:57:37.661 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/37 file after the snapshot 2024-03-19T02:57:37.663 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/38 file after the snapshot 2024-03-19T02:57:37.664 INFO:tasks.workunit.client.0.smithi092.stdout: create 211/39 file after the snapshot 2024-03-19T02:57:37.666 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/.snap/snap-subdir-test 2024-03-19T02:57:37.697 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/30 file after the snapshot 2024-03-19T02:57:37.698 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/31 file after the snapshot 2024-03-19T02:57:37.700 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/32 file after the snapshot 2024-03-19T02:57:37.701 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/33 file after the snapshot 2024-03-19T02:57:37.703 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/34 file after the snapshot 2024-03-19T02:57:37.704 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/35 file after the snapshot 2024-03-19T02:57:37.705 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/36 file after the snapshot 2024-03-19T02:57:37.707 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/37 file after the snapshot 2024-03-19T02:57:37.708 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/38 file after the snapshot 2024-03-19T02:57:37.710 INFO:tasks.workunit.client.0.smithi092.stdout: create 212/39 file after the snapshot 2024-03-19T02:57:37.711 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/.snap/snap-subdir-test 2024-03-19T02:57:37.741 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/30 file after the snapshot 2024-03-19T02:57:37.742 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/31 file after the snapshot 2024-03-19T02:57:37.744 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/32 file after the snapshot 2024-03-19T02:57:37.745 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/33 file after the snapshot 2024-03-19T02:57:37.746 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/34 file after the snapshot 2024-03-19T02:57:37.748 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/35 file after the snapshot 2024-03-19T02:57:37.749 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/36 file after the snapshot 2024-03-19T02:57:37.751 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/37 file after the snapshot 2024-03-19T02:57:37.752 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/38 file after the snapshot 2024-03-19T02:57:37.754 INFO:tasks.workunit.client.0.smithi092.stdout: create 213/39 file after the snapshot 2024-03-19T02:57:37.755 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/.snap/snap-subdir-test 2024-03-19T02:57:37.784 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/30 file after the snapshot 2024-03-19T02:57:37.785 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/31 file after the snapshot 2024-03-19T02:57:37.786 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/32 file after the snapshot 2024-03-19T02:57:37.788 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/33 file after the snapshot 2024-03-19T02:57:37.789 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/34 file after the snapshot 2024-03-19T02:57:37.790 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/35 file after the snapshot 2024-03-19T02:57:37.792 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/36 file after the snapshot 2024-03-19T02:57:37.794 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/37 file after the snapshot 2024-03-19T02:57:37.795 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/38 file after the snapshot 2024-03-19T02:57:37.796 INFO:tasks.workunit.client.0.smithi092.stdout: create 214/39 file after the snapshot 2024-03-19T02:57:37.798 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/.snap/snap-subdir-test 2024-03-19T02:57:37.829 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/30 file after the snapshot 2024-03-19T02:57:37.830 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/31 file after the snapshot 2024-03-19T02:57:37.833 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/32 file after the snapshot 2024-03-19T02:57:37.834 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/33 file after the snapshot 2024-03-19T02:57:37.834 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/34 file after the snapshot 2024-03-19T02:57:37.836 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/35 file after the snapshot 2024-03-19T02:57:37.837 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/36 file after the snapshot 2024-03-19T02:57:37.839 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/37 file after the snapshot 2024-03-19T02:57:37.840 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/38 file after the snapshot 2024-03-19T02:57:37.842 INFO:tasks.workunit.client.0.smithi092.stdout: create 215/39 file after the snapshot 2024-03-19T02:57:37.843 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/.snap/snap-subdir-test 2024-03-19T02:57:37.875 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/30 file after the snapshot 2024-03-19T02:57:37.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/31 file after the snapshot 2024-03-19T02:57:37.878 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/32 file after the snapshot 2024-03-19T02:57:37.879 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/33 file after the snapshot 2024-03-19T02:57:37.881 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/34 file after the snapshot 2024-03-19T02:57:37.882 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/35 file after the snapshot 2024-03-19T02:57:37.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/36 file after the snapshot 2024-03-19T02:57:37.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/37 file after the snapshot 2024-03-19T02:57:37.886 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/38 file after the snapshot 2024-03-19T02:57:37.887 INFO:tasks.workunit.client.0.smithi092.stdout: create 216/39 file after the snapshot 2024-03-19T02:57:37.889 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/.snap/snap-subdir-test 2024-03-19T02:57:37.920 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/30 file after the snapshot 2024-03-19T02:57:37.922 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/31 file after the snapshot 2024-03-19T02:57:37.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/32 file after the snapshot 2024-03-19T02:57:37.925 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/33 file after the snapshot 2024-03-19T02:57:37.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/34 file after the snapshot 2024-03-19T02:57:37.928 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/35 file after the snapshot 2024-03-19T02:57:37.929 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/36 file after the snapshot 2024-03-19T02:57:37.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/37 file after the snapshot 2024-03-19T02:57:37.932 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/38 file after the snapshot 2024-03-19T02:57:37.933 INFO:tasks.workunit.client.0.smithi092.stdout: create 217/39 file after the snapshot 2024-03-19T02:57:37.935 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/.snap/snap-subdir-test 2024-03-19T02:57:37.967 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/30 file after the snapshot 2024-03-19T02:57:37.968 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/31 file after the snapshot 2024-03-19T02:57:37.969 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/32 file after the snapshot 2024-03-19T02:57:37.971 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/33 file after the snapshot 2024-03-19T02:57:37.972 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/34 file after the snapshot 2024-03-19T02:57:37.973 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/35 file after the snapshot 2024-03-19T02:57:37.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/36 file after the snapshot 2024-03-19T02:57:37.976 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/37 file after the snapshot 2024-03-19T02:57:37.977 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/38 file after the snapshot 2024-03-19T02:57:37.979 INFO:tasks.workunit.client.0.smithi092.stdout: create 218/39 file after the snapshot 2024-03-19T02:57:37.980 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/.snap/snap-subdir-test 2024-03-19T02:57:38.010 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/30 file after the snapshot 2024-03-19T02:57:38.012 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/31 file after the snapshot 2024-03-19T02:57:38.013 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/32 file after the snapshot 2024-03-19T02:57:38.014 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/33 file after the snapshot 2024-03-19T02:57:38.016 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/34 file after the snapshot 2024-03-19T02:57:38.017 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/35 file after the snapshot 2024-03-19T02:57:38.018 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/36 file after the snapshot 2024-03-19T02:57:38.019 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/37 file after the snapshot 2024-03-19T02:57:38.021 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/38 file after the snapshot 2024-03-19T02:57:38.022 INFO:tasks.workunit.client.0.smithi092.stdout: create 219/39 file after the snapshot 2024-03-19T02:57:38.023 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/.snap/snap-subdir-test 2024-03-19T02:57:38.053 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/30 file after the snapshot 2024-03-19T02:57:38.054 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/31 file after the snapshot 2024-03-19T02:57:38.055 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/32 file after the snapshot 2024-03-19T02:57:38.057 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/33 file after the snapshot 2024-03-19T02:57:38.058 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/34 file after the snapshot 2024-03-19T02:57:38.060 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/35 file after the snapshot 2024-03-19T02:57:38.061 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/36 file after the snapshot 2024-03-19T02:57:38.063 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/37 file after the snapshot 2024-03-19T02:57:38.064 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/38 file after the snapshot 2024-03-19T02:57:38.066 INFO:tasks.workunit.client.0.smithi092.stdout: create 220/39 file after the snapshot 2024-03-19T02:57:38.067 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/.snap/snap-subdir-test 2024-03-19T02:57:38.097 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/30 file after the snapshot 2024-03-19T02:57:38.098 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/31 file after the snapshot 2024-03-19T02:57:38.100 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/32 file after the snapshot 2024-03-19T02:57:38.101 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/33 file after the snapshot 2024-03-19T02:57:38.103 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/34 file after the snapshot 2024-03-19T02:57:38.104 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/35 file after the snapshot 2024-03-19T02:57:38.106 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/36 file after the snapshot 2024-03-19T02:57:38.107 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/37 file after the snapshot 2024-03-19T02:57:38.108 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/38 file after the snapshot 2024-03-19T02:57:38.110 INFO:tasks.workunit.client.0.smithi092.stdout: create 221/39 file after the snapshot 2024-03-19T02:57:38.111 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/.snap/snap-subdir-test 2024-03-19T02:57:38.143 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/30 file after the snapshot 2024-03-19T02:57:38.144 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/31 file after the snapshot 2024-03-19T02:57:38.145 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/32 file after the snapshot 2024-03-19T02:57:38.147 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/33 file after the snapshot 2024-03-19T02:57:38.148 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/34 file after the snapshot 2024-03-19T02:57:38.149 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/35 file after the snapshot 2024-03-19T02:57:38.151 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/36 file after the snapshot 2024-03-19T02:57:38.152 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/37 file after the snapshot 2024-03-19T02:57:38.154 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/38 file after the snapshot 2024-03-19T02:57:38.155 INFO:tasks.workunit.client.0.smithi092.stdout: create 222/39 file after the snapshot 2024-03-19T02:57:38.157 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/.snap/snap-subdir-test 2024-03-19T02:57:38.189 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/30 file after the snapshot 2024-03-19T02:57:38.191 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/31 file after the snapshot 2024-03-19T02:57:38.192 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/32 file after the snapshot 2024-03-19T02:57:38.194 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/33 file after the snapshot 2024-03-19T02:57:38.195 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/34 file after the snapshot 2024-03-19T02:57:38.197 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/35 file after the snapshot 2024-03-19T02:57:38.198 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/36 file after the snapshot 2024-03-19T02:57:38.199 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/37 file after the snapshot 2024-03-19T02:57:38.201 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/38 file after the snapshot 2024-03-19T02:57:38.202 INFO:tasks.workunit.client.0.smithi092.stdout: create 223/39 file after the snapshot 2024-03-19T02:57:38.204 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/.snap/snap-subdir-test 2024-03-19T02:57:38.235 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/30 file after the snapshot 2024-03-19T02:57:38.236 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/31 file after the snapshot 2024-03-19T02:57:38.237 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/32 file after the snapshot 2024-03-19T02:57:38.239 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/33 file after the snapshot 2024-03-19T02:57:38.240 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/34 file after the snapshot 2024-03-19T02:57:38.241 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/35 file after the snapshot 2024-03-19T02:57:38.242 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/36 file after the snapshot 2024-03-19T02:57:38.244 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/37 file after the snapshot 2024-03-19T02:57:38.245 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/38 file after the snapshot 2024-03-19T02:57:38.246 INFO:tasks.workunit.client.0.smithi092.stdout: create 224/39 file after the snapshot 2024-03-19T02:57:38.247 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/.snap/snap-subdir-test 2024-03-19T02:57:38.278 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/30 file after the snapshot 2024-03-19T02:57:38.279 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/31 file after the snapshot 2024-03-19T02:57:38.281 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/32 file after the snapshot 2024-03-19T02:57:38.282 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/33 file after the snapshot 2024-03-19T02:57:38.283 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/34 file after the snapshot 2024-03-19T02:57:38.285 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/35 file after the snapshot 2024-03-19T02:57:38.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/36 file after the snapshot 2024-03-19T02:57:38.288 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/37 file after the snapshot 2024-03-19T02:57:38.289 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/38 file after the snapshot 2024-03-19T02:57:38.290 INFO:tasks.workunit.client.0.smithi092.stdout: create 225/39 file after the snapshot 2024-03-19T02:57:38.292 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/.snap/snap-subdir-test 2024-03-19T02:57:38.321 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/30 file after the snapshot 2024-03-19T02:57:38.323 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/31 file after the snapshot 2024-03-19T02:57:38.324 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/32 file after the snapshot 2024-03-19T02:57:38.326 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/33 file after the snapshot 2024-03-19T02:57:38.327 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/34 file after the snapshot 2024-03-19T02:57:38.328 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/35 file after the snapshot 2024-03-19T02:57:38.330 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/36 file after the snapshot 2024-03-19T02:57:38.331 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/37 file after the snapshot 2024-03-19T02:57:38.333 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/38 file after the snapshot 2024-03-19T02:57:38.334 INFO:tasks.workunit.client.0.smithi092.stdout: create 226/39 file after the snapshot 2024-03-19T02:57:38.336 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/.snap/snap-subdir-test 2024-03-19T02:57:38.367 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/30 file after the snapshot 2024-03-19T02:57:38.369 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/31 file after the snapshot 2024-03-19T02:57:38.371 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/32 file after the snapshot 2024-03-19T02:57:38.373 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/33 file after the snapshot 2024-03-19T02:57:38.375 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/34 file after the snapshot 2024-03-19T02:57:38.377 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/35 file after the snapshot 2024-03-19T02:57:38.379 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/36 file after the snapshot 2024-03-19T02:57:38.381 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/37 file after the snapshot 2024-03-19T02:57:38.383 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/38 file after the snapshot 2024-03-19T02:57:38.385 INFO:tasks.workunit.client.0.smithi092.stdout: create 227/39 file after the snapshot 2024-03-19T02:57:38.387 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/.snap/snap-subdir-test 2024-03-19T02:57:38.418 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/30 file after the snapshot 2024-03-19T02:57:38.420 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/31 file after the snapshot 2024-03-19T02:57:38.422 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/32 file after the snapshot 2024-03-19T02:57:38.424 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/33 file after the snapshot 2024-03-19T02:57:38.426 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/34 file after the snapshot 2024-03-19T02:57:38.428 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/35 file after the snapshot 2024-03-19T02:57:38.430 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/36 file after the snapshot 2024-03-19T02:57:38.432 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/37 file after the snapshot 2024-03-19T02:57:38.434 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/38 file after the snapshot 2024-03-19T02:57:38.436 INFO:tasks.workunit.client.0.smithi092.stdout: create 228/39 file after the snapshot 2024-03-19T02:57:38.438 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/.snap/snap-subdir-test 2024-03-19T02:57:38.471 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/30 file after the snapshot 2024-03-19T02:57:38.472 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/31 file after the snapshot 2024-03-19T02:57:38.474 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/32 file after the snapshot 2024-03-19T02:57:38.475 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/33 file after the snapshot 2024-03-19T02:57:38.477 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/34 file after the snapshot 2024-03-19T02:57:38.478 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/35 file after the snapshot 2024-03-19T02:57:38.480 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/36 file after the snapshot 2024-03-19T02:57:38.481 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/37 file after the snapshot 2024-03-19T02:57:38.482 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/38 file after the snapshot 2024-03-19T02:57:38.484 INFO:tasks.workunit.client.0.smithi092.stdout: create 229/39 file after the snapshot 2024-03-19T02:57:38.485 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/.snap/snap-subdir-test 2024-03-19T02:57:38.517 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/30 file after the snapshot 2024-03-19T02:57:38.519 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/31 file after the snapshot 2024-03-19T02:57:38.521 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/32 file after the snapshot 2024-03-19T02:57:38.523 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/33 file after the snapshot 2024-03-19T02:57:38.525 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/34 file after the snapshot 2024-03-19T02:57:38.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/35 file after the snapshot 2024-03-19T02:57:38.528 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/36 file after the snapshot 2024-03-19T02:57:38.530 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/37 file after the snapshot 2024-03-19T02:57:38.532 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/38 file after the snapshot 2024-03-19T02:57:38.533 INFO:tasks.workunit.client.0.smithi092.stdout: create 230/39 file after the snapshot 2024-03-19T02:57:38.535 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/.snap/snap-subdir-test 2024-03-19T02:57:38.568 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/30 file after the snapshot 2024-03-19T02:57:38.570 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/31 file after the snapshot 2024-03-19T02:57:38.571 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/32 file after the snapshot 2024-03-19T02:57:38.573 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/33 file after the snapshot 2024-03-19T02:57:38.574 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/34 file after the snapshot 2024-03-19T02:57:38.576 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/35 file after the snapshot 2024-03-19T02:57:38.577 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/36 file after the snapshot 2024-03-19T02:57:38.864 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/37 file after the snapshot 2024-03-19T02:57:38.884 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/38 file after the snapshot 2024-03-19T02:57:38.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 231/39 file after the snapshot 2024-03-19T02:57:38.887 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/.snap/snap-subdir-test 2024-03-19T02:57:38.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/30 file after the snapshot 2024-03-19T02:57:38.924 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/31 file after the snapshot 2024-03-19T02:57:38.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/32 file after the snapshot 2024-03-19T02:57:38.927 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/33 file after the snapshot 2024-03-19T02:57:38.929 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/34 file after the snapshot 2024-03-19T02:57:38.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/35 file after the snapshot 2024-03-19T02:57:38.932 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/36 file after the snapshot 2024-03-19T02:57:38.933 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/37 file after the snapshot 2024-03-19T02:57:38.935 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/38 file after the snapshot 2024-03-19T02:57:38.936 INFO:tasks.workunit.client.0.smithi092.stdout: create 232/39 file after the snapshot 2024-03-19T02:57:38.938 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/.snap/snap-subdir-test 2024-03-19T02:57:38.970 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/30 file after the snapshot 2024-03-19T02:57:38.972 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/31 file after the snapshot 2024-03-19T02:57:38.974 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/32 file after the snapshot 2024-03-19T02:57:38.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/33 file after the snapshot 2024-03-19T02:57:38.976 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/34 file after the snapshot 2024-03-19T02:57:38.978 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/35 file after the snapshot 2024-03-19T02:57:38.980 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/36 file after the snapshot 2024-03-19T02:57:38.981 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/37 file after the snapshot 2024-03-19T02:57:38.983 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/38 file after the snapshot 2024-03-19T02:57:38.984 INFO:tasks.workunit.client.0.smithi092.stdout: create 233/39 file after the snapshot 2024-03-19T02:57:38.986 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/.snap/snap-subdir-test 2024-03-19T02:57:39.015 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/30 file after the snapshot 2024-03-19T02:57:39.016 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/31 file after the snapshot 2024-03-19T02:57:39.018 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/32 file after the snapshot 2024-03-19T02:57:39.019 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/33 file after the snapshot 2024-03-19T02:57:39.021 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/34 file after the snapshot 2024-03-19T02:57:39.022 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/35 file after the snapshot 2024-03-19T02:57:39.024 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/36 file after the snapshot 2024-03-19T02:57:39.025 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/37 file after the snapshot 2024-03-19T02:57:39.026 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/38 file after the snapshot 2024-03-19T02:57:39.028 INFO:tasks.workunit.client.0.smithi092.stdout: create 234/39 file after the snapshot 2024-03-19T02:57:39.029 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/.snap/snap-subdir-test 2024-03-19T02:57:40.028 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/30 file after the snapshot 2024-03-19T02:57:40.029 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/31 file after the snapshot 2024-03-19T02:57:40.031 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/32 file after the snapshot 2024-03-19T02:57:40.033 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/33 file after the snapshot 2024-03-19T02:57:40.034 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/34 file after the snapshot 2024-03-19T02:57:40.035 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/35 file after the snapshot 2024-03-19T02:57:40.038 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/36 file after the snapshot 2024-03-19T02:57:40.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/37 file after the snapshot 2024-03-19T02:57:40.041 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/38 file after the snapshot 2024-03-19T02:57:40.042 INFO:tasks.workunit.client.0.smithi092.stdout: create 235/39 file after the snapshot 2024-03-19T02:57:40.044 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/.snap/snap-subdir-test 2024-03-19T02:57:40.075 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/30 file after the snapshot 2024-03-19T02:57:40.077 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/31 file after the snapshot 2024-03-19T02:57:40.078 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/32 file after the snapshot 2024-03-19T02:57:40.080 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/33 file after the snapshot 2024-03-19T02:57:40.081 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/34 file after the snapshot 2024-03-19T02:57:40.083 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/35 file after the snapshot 2024-03-19T02:57:40.085 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/36 file after the snapshot 2024-03-19T02:57:40.086 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/37 file after the snapshot 2024-03-19T02:57:40.088 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/38 file after the snapshot 2024-03-19T02:57:40.089 INFO:tasks.workunit.client.0.smithi092.stdout: create 236/39 file after the snapshot 2024-03-19T02:57:40.090 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/.snap/snap-subdir-test 2024-03-19T02:57:40.122 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/30 file after the snapshot 2024-03-19T02:57:40.123 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/31 file after the snapshot 2024-03-19T02:57:40.124 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/32 file after the snapshot 2024-03-19T02:57:40.126 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/33 file after the snapshot 2024-03-19T02:57:40.127 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/34 file after the snapshot 2024-03-19T02:57:40.129 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/35 file after the snapshot 2024-03-19T02:57:40.130 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/36 file after the snapshot 2024-03-19T02:57:40.131 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/37 file after the snapshot 2024-03-19T02:57:40.134 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/38 file after the snapshot 2024-03-19T02:57:40.135 INFO:tasks.workunit.client.0.smithi092.stdout: create 237/39 file after the snapshot 2024-03-19T02:57:40.137 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/.snap/snap-subdir-test 2024-03-19T02:57:40.168 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/30 file after the snapshot 2024-03-19T02:57:40.170 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/31 file after the snapshot 2024-03-19T02:57:40.171 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/32 file after the snapshot 2024-03-19T02:57:40.172 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/33 file after the snapshot 2024-03-19T02:57:40.174 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/34 file after the snapshot 2024-03-19T02:57:40.175 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/35 file after the snapshot 2024-03-19T02:57:40.176 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/36 file after the snapshot 2024-03-19T02:57:40.178 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/37 file after the snapshot 2024-03-19T02:57:40.179 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/38 file after the snapshot 2024-03-19T02:57:40.181 INFO:tasks.workunit.client.0.smithi092.stdout: create 238/39 file after the snapshot 2024-03-19T02:57:40.182 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/.snap/snap-subdir-test 2024-03-19T02:57:40.218 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/30 file after the snapshot 2024-03-19T02:57:40.219 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/31 file after the snapshot 2024-03-19T02:57:40.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/32 file after the snapshot 2024-03-19T02:57:40.222 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/33 file after the snapshot 2024-03-19T02:57:40.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/34 file after the snapshot 2024-03-19T02:57:40.225 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/35 file after the snapshot 2024-03-19T02:57:40.227 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/36 file after the snapshot 2024-03-19T02:57:40.228 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/37 file after the snapshot 2024-03-19T02:57:40.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/38 file after the snapshot 2024-03-19T02:57:40.231 INFO:tasks.workunit.client.0.smithi092.stdout: create 239/39 file after the snapshot 2024-03-19T02:57:40.233 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/.snap/snap-subdir-test 2024-03-19T02:57:40.266 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/30 file after the snapshot 2024-03-19T02:57:40.268 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/31 file after the snapshot 2024-03-19T02:57:40.270 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/32 file after the snapshot 2024-03-19T02:57:40.272 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/33 file after the snapshot 2024-03-19T02:57:40.274 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/34 file after the snapshot 2024-03-19T02:57:40.276 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/35 file after the snapshot 2024-03-19T02:57:40.278 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/36 file after the snapshot 2024-03-19T02:57:40.280 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/37 file after the snapshot 2024-03-19T02:57:40.282 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/38 file after the snapshot 2024-03-19T02:57:40.284 INFO:tasks.workunit.client.0.smithi092.stdout: create 240/39 file after the snapshot 2024-03-19T02:57:40.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/.snap/snap-subdir-test 2024-03-19T02:57:40.322 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/30 file after the snapshot 2024-03-19T02:57:40.323 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/31 file after the snapshot 2024-03-19T02:57:40.324 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/32 file after the snapshot 2024-03-19T02:57:40.326 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/33 file after the snapshot 2024-03-19T02:57:40.328 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/34 file after the snapshot 2024-03-19T02:57:40.329 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/35 file after the snapshot 2024-03-19T02:57:40.331 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/36 file after the snapshot 2024-03-19T02:57:40.332 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/37 file after the snapshot 2024-03-19T02:57:40.334 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/38 file after the snapshot 2024-03-19T02:57:40.336 INFO:tasks.workunit.client.0.smithi092.stdout: create 241/39 file after the snapshot 2024-03-19T02:57:40.337 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/.snap/snap-subdir-test 2024-03-19T02:57:40.372 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/30 file after the snapshot 2024-03-19T02:57:40.374 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/31 file after the snapshot 2024-03-19T02:57:40.375 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/32 file after the snapshot 2024-03-19T02:57:40.377 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/33 file after the snapshot 2024-03-19T02:57:40.378 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/34 file after the snapshot 2024-03-19T02:57:40.380 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/35 file after the snapshot 2024-03-19T02:57:40.911 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/36 file after the snapshot 2024-03-19T02:57:41.185 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/37 file after the snapshot 2024-03-19T02:57:41.203 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/38 file after the snapshot 2024-03-19T02:57:41.232 INFO:tasks.workunit.client.0.smithi092.stdout: create 242/39 file after the snapshot 2024-03-19T02:57:41.527 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/.snap/snap-subdir-test 2024-03-19T02:57:41.972 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/30 file after the snapshot 2024-03-19T02:57:41.973 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/31 file after the snapshot 2024-03-19T02:57:41.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/32 file after the snapshot 2024-03-19T02:57:41.977 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/33 file after the snapshot 2024-03-19T02:57:41.979 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/34 file after the snapshot 2024-03-19T02:57:41.981 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/35 file after the snapshot 2024-03-19T02:57:41.983 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/36 file after the snapshot 2024-03-19T02:57:41.984 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/37 file after the snapshot 2024-03-19T02:57:41.986 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/38 file after the snapshot 2024-03-19T02:57:41.988 INFO:tasks.workunit.client.0.smithi092.stdout: create 243/39 file after the snapshot 2024-03-19T02:57:41.990 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/.snap/snap-subdir-test 2024-03-19T02:57:42.033 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/30 file after the snapshot 2024-03-19T02:57:42.035 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/31 file after the snapshot 2024-03-19T02:57:42.036 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/32 file after the snapshot 2024-03-19T02:57:42.038 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/33 file after the snapshot 2024-03-19T02:57:42.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/34 file after the snapshot 2024-03-19T02:57:42.042 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/35 file after the snapshot 2024-03-19T02:57:42.044 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/36 file after the snapshot 2024-03-19T02:57:42.046 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/37 file after the snapshot 2024-03-19T02:57:42.048 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/38 file after the snapshot 2024-03-19T02:57:42.049 INFO:tasks.workunit.client.0.smithi092.stdout: create 244/39 file after the snapshot 2024-03-19T02:57:42.052 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/.snap/snap-subdir-test 2024-03-19T02:57:42.095 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/30 file after the snapshot 2024-03-19T02:57:42.098 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/31 file after the snapshot 2024-03-19T02:57:42.100 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/32 file after the snapshot 2024-03-19T02:57:42.102 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/33 file after the snapshot 2024-03-19T02:57:42.104 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/34 file after the snapshot 2024-03-19T02:57:42.106 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/35 file after the snapshot 2024-03-19T02:57:42.107 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/36 file after the snapshot 2024-03-19T02:57:42.108 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/37 file after the snapshot 2024-03-19T02:57:42.110 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/38 file after the snapshot 2024-03-19T02:57:42.111 INFO:tasks.workunit.client.0.smithi092.stdout: create 245/39 file after the snapshot 2024-03-19T02:57:42.113 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/.snap/snap-subdir-test 2024-03-19T02:57:42.163 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/30 file after the snapshot 2024-03-19T02:57:42.165 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/31 file after the snapshot 2024-03-19T02:57:42.167 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/32 file after the snapshot 2024-03-19T02:57:42.168 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/33 file after the snapshot 2024-03-19T02:57:42.170 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/34 file after the snapshot 2024-03-19T02:57:42.171 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/35 file after the snapshot 2024-03-19T02:57:42.172 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/36 file after the snapshot 2024-03-19T02:57:42.174 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/37 file after the snapshot 2024-03-19T02:57:42.175 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/38 file after the snapshot 2024-03-19T02:57:42.177 INFO:tasks.workunit.client.0.smithi092.stdout: create 246/39 file after the snapshot 2024-03-19T02:57:42.178 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/.snap/snap-subdir-test 2024-03-19T02:57:42.221 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/30 file after the snapshot 2024-03-19T02:57:42.223 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/31 file after the snapshot 2024-03-19T02:57:42.224 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/32 file after the snapshot 2024-03-19T02:57:42.226 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/33 file after the snapshot 2024-03-19T02:57:42.227 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/34 file after the snapshot 2024-03-19T02:57:42.229 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/35 file after the snapshot 2024-03-19T02:57:42.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/36 file after the snapshot 2024-03-19T02:57:42.231 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/37 file after the snapshot 2024-03-19T02:57:42.233 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/38 file after the snapshot 2024-03-19T02:57:42.234 INFO:tasks.workunit.client.0.smithi092.stdout: create 247/39 file after the snapshot 2024-03-19T02:57:42.235 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/.snap/snap-subdir-test 2024-03-19T02:57:42.281 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/30 file after the snapshot 2024-03-19T02:57:42.282 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/31 file after the snapshot 2024-03-19T02:57:42.283 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/32 file after the snapshot 2024-03-19T02:57:42.285 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/33 file after the snapshot 2024-03-19T02:57:42.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/34 file after the snapshot 2024-03-19T02:57:42.288 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/35 file after the snapshot 2024-03-19T02:57:42.289 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/36 file after the snapshot 2024-03-19T02:57:42.291 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/37 file after the snapshot 2024-03-19T02:57:42.292 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/38 file after the snapshot 2024-03-19T02:57:42.293 INFO:tasks.workunit.client.0.smithi092.stdout: create 248/39 file after the snapshot 2024-03-19T02:57:42.295 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/.snap/snap-subdir-test 2024-03-19T02:57:42.341 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/30 file after the snapshot 2024-03-19T02:57:42.342 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/31 file after the snapshot 2024-03-19T02:57:42.344 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/32 file after the snapshot 2024-03-19T02:57:42.345 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/33 file after the snapshot 2024-03-19T02:57:42.347 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/34 file after the snapshot 2024-03-19T02:57:42.348 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/35 file after the snapshot 2024-03-19T02:57:42.350 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/36 file after the snapshot 2024-03-19T02:57:42.354 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/37 file after the snapshot 2024-03-19T02:57:42.354 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/38 file after the snapshot 2024-03-19T02:57:42.354 INFO:tasks.workunit.client.0.smithi092.stdout: create 249/39 file after the snapshot 2024-03-19T02:57:42.355 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/.snap/snap-subdir-test 2024-03-19T02:57:42.401 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/30 file after the snapshot 2024-03-19T02:57:42.402 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/31 file after the snapshot 2024-03-19T02:57:42.403 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/32 file after the snapshot 2024-03-19T02:57:42.405 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/33 file after the snapshot 2024-03-19T02:57:42.406 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/34 file after the snapshot 2024-03-19T02:57:42.407 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/35 file after the snapshot 2024-03-19T02:57:42.408 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/36 file after the snapshot 2024-03-19T02:57:42.409 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/37 file after the snapshot 2024-03-19T02:57:42.410 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/38 file after the snapshot 2024-03-19T02:57:42.412 INFO:tasks.workunit.client.0.smithi092.stdout: create 250/39 file after the snapshot 2024-03-19T02:57:42.413 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/.snap/snap-subdir-test 2024-03-19T02:57:42.457 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/30 file after the snapshot 2024-03-19T02:57:42.459 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/31 file after the snapshot 2024-03-19T02:57:42.460 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/32 file after the snapshot 2024-03-19T02:57:42.461 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/33 file after the snapshot 2024-03-19T02:57:42.463 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/34 file after the snapshot 2024-03-19T02:57:42.464 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/35 file after the snapshot 2024-03-19T02:57:42.466 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/36 file after the snapshot 2024-03-19T02:57:42.467 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/37 file after the snapshot 2024-03-19T02:57:42.468 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/38 file after the snapshot 2024-03-19T02:57:42.470 INFO:tasks.workunit.client.0.smithi092.stdout: create 251/39 file after the snapshot 2024-03-19T02:57:42.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/.snap/snap-subdir-test 2024-03-19T02:57:42.570 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/30 file after the snapshot 2024-03-19T02:57:42.572 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/31 file after the snapshot 2024-03-19T02:57:42.577 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/32 file after the snapshot 2024-03-19T02:57:42.578 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/33 file after the snapshot 2024-03-19T02:57:42.579 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/34 file after the snapshot 2024-03-19T02:57:42.579 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/35 file after the snapshot 2024-03-19T02:57:42.580 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/36 file after the snapshot 2024-03-19T02:57:42.582 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/37 file after the snapshot 2024-03-19T02:57:42.585 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/38 file after the snapshot 2024-03-19T02:57:42.587 INFO:tasks.workunit.client.0.smithi092.stdout: create 252/39 file after the snapshot 2024-03-19T02:57:42.588 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/.snap/snap-subdir-test 2024-03-19T02:57:42.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/30 file after the snapshot 2024-03-19T02:57:42.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/31 file after the snapshot 2024-03-19T02:57:42.624 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/32 file after the snapshot 2024-03-19T02:57:42.625 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/33 file after the snapshot 2024-03-19T02:57:42.627 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/34 file after the snapshot 2024-03-19T02:57:42.629 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/35 file after the snapshot 2024-03-19T02:57:42.634 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/36 file after the snapshot 2024-03-19T02:57:42.635 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/37 file after the snapshot 2024-03-19T02:57:42.635 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/38 file after the snapshot 2024-03-19T02:57:42.637 INFO:tasks.workunit.client.0.smithi092.stdout: create 253/39 file after the snapshot 2024-03-19T02:57:42.640 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/.snap/snap-subdir-test 2024-03-19T02:57:42.671 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/30 file after the snapshot 2024-03-19T02:57:42.674 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/31 file after the snapshot 2024-03-19T02:57:42.675 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/32 file after the snapshot 2024-03-19T02:57:42.677 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/33 file after the snapshot 2024-03-19T02:57:42.679 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/34 file after the snapshot 2024-03-19T02:57:42.681 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/35 file after the snapshot 2024-03-19T02:57:42.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/36 file after the snapshot 2024-03-19T02:57:42.684 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/37 file after the snapshot 2024-03-19T02:57:42.686 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/38 file after the snapshot 2024-03-19T02:57:42.688 INFO:tasks.workunit.client.0.smithi092.stdout: create 254/39 file after the snapshot 2024-03-19T02:57:42.690 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/.snap/snap-subdir-test 2024-03-19T02:57:42.723 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/30 file after the snapshot 2024-03-19T02:57:42.724 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/31 file after the snapshot 2024-03-19T02:57:42.726 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/32 file after the snapshot 2024-03-19T02:57:42.728 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/33 file after the snapshot 2024-03-19T02:57:42.730 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/34 file after the snapshot 2024-03-19T02:57:42.732 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/35 file after the snapshot 2024-03-19T02:57:42.734 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/36 file after the snapshot 2024-03-19T02:57:42.736 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/37 file after the snapshot 2024-03-19T02:57:42.737 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/38 file after the snapshot 2024-03-19T02:57:42.739 INFO:tasks.workunit.client.0.smithi092.stdout: create 255/39 file after the snapshot 2024-03-19T02:57:42.741 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/.snap/snap-subdir-test 2024-03-19T02:57:42.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/30 file after the snapshot 2024-03-19T02:57:42.775 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/31 file after the snapshot 2024-03-19T02:57:42.777 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/32 file after the snapshot 2024-03-19T02:57:42.779 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/33 file after the snapshot 2024-03-19T02:57:42.781 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/34 file after the snapshot 2024-03-19T02:57:42.783 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/35 file after the snapshot 2024-03-19T02:57:42.785 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/36 file after the snapshot 2024-03-19T02:57:42.787 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/37 file after the snapshot 2024-03-19T02:57:42.789 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/38 file after the snapshot 2024-03-19T02:57:42.791 INFO:tasks.workunit.client.0.smithi092.stdout: create 256/39 file after the snapshot 2024-03-19T02:57:42.793 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/.snap/snap-subdir-test 2024-03-19T02:57:42.825 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/30 file after the snapshot 2024-03-19T02:57:42.827 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/31 file after the snapshot 2024-03-19T02:57:42.828 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/32 file after the snapshot 2024-03-19T02:57:42.829 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/33 file after the snapshot 2024-03-19T02:57:42.831 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/34 file after the snapshot 2024-03-19T02:57:42.832 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/35 file after the snapshot 2024-03-19T02:57:42.833 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/36 file after the snapshot 2024-03-19T02:57:42.835 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/37 file after the snapshot 2024-03-19T02:57:42.836 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/38 file after the snapshot 2024-03-19T02:57:42.837 INFO:tasks.workunit.client.0.smithi092.stdout: create 257/39 file after the snapshot 2024-03-19T02:57:42.839 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/.snap/snap-subdir-test 2024-03-19T02:57:42.872 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/30 file after the snapshot 2024-03-19T02:57:42.873 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/31 file after the snapshot 2024-03-19T02:57:42.874 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/32 file after the snapshot 2024-03-19T02:57:42.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/33 file after the snapshot 2024-03-19T02:57:42.877 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/34 file after the snapshot 2024-03-19T02:57:42.879 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/35 file after the snapshot 2024-03-19T02:57:42.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/36 file after the snapshot 2024-03-19T02:57:42.881 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/37 file after the snapshot 2024-03-19T02:57:42.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/38 file after the snapshot 2024-03-19T02:57:42.884 INFO:tasks.workunit.client.0.smithi092.stdout: create 258/39 file after the snapshot 2024-03-19T02:57:42.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/.snap/snap-subdir-test 2024-03-19T02:57:42.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/30 file after the snapshot 2024-03-19T02:57:42.921 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/31 file after the snapshot 2024-03-19T02:57:42.922 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/32 file after the snapshot 2024-03-19T02:57:42.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/33 file after the snapshot 2024-03-19T02:57:42.925 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/34 file after the snapshot 2024-03-19T02:57:42.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/35 file after the snapshot 2024-03-19T02:57:42.927 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/36 file after the snapshot 2024-03-19T02:57:42.929 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/37 file after the snapshot 2024-03-19T02:57:42.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/38 file after the snapshot 2024-03-19T02:57:42.931 INFO:tasks.workunit.client.0.smithi092.stdout: create 259/39 file after the snapshot 2024-03-19T02:57:42.932 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/.snap/snap-subdir-test 2024-03-19T02:57:42.963 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/30 file after the snapshot 2024-03-19T02:57:42.965 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/31 file after the snapshot 2024-03-19T02:57:42.966 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/32 file after the snapshot 2024-03-19T02:57:42.968 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/33 file after the snapshot 2024-03-19T02:57:42.969 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/34 file after the snapshot 2024-03-19T02:57:42.970 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/35 file after the snapshot 2024-03-19T02:57:42.972 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/36 file after the snapshot 2024-03-19T02:57:42.973 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/37 file after the snapshot 2024-03-19T02:57:42.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/38 file after the snapshot 2024-03-19T02:57:42.976 INFO:tasks.workunit.client.0.smithi092.stdout: create 260/39 file after the snapshot 2024-03-19T02:57:42.978 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/.snap/snap-subdir-test 2024-03-19T02:57:43.011 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/30 file after the snapshot 2024-03-19T02:57:43.012 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/31 file after the snapshot 2024-03-19T02:57:43.014 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/32 file after the snapshot 2024-03-19T02:57:43.015 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/33 file after the snapshot 2024-03-19T02:57:43.016 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/34 file after the snapshot 2024-03-19T02:57:43.018 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/35 file after the snapshot 2024-03-19T02:57:43.019 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/36 file after the snapshot 2024-03-19T02:57:43.021 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/37 file after the snapshot 2024-03-19T02:57:43.022 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/38 file after the snapshot 2024-03-19T02:57:43.023 INFO:tasks.workunit.client.0.smithi092.stdout: create 261/39 file after the snapshot 2024-03-19T02:57:43.025 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/.snap/snap-subdir-test 2024-03-19T02:57:43.057 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/30 file after the snapshot 2024-03-19T02:57:43.059 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/31 file after the snapshot 2024-03-19T02:57:43.060 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/32 file after the snapshot 2024-03-19T02:57:43.062 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/33 file after the snapshot 2024-03-19T02:57:43.063 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/34 file after the snapshot 2024-03-19T02:57:43.065 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/35 file after the snapshot 2024-03-19T02:57:43.066 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/36 file after the snapshot 2024-03-19T02:57:43.068 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/37 file after the snapshot 2024-03-19T02:57:43.069 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/38 file after the snapshot 2024-03-19T02:57:43.070 INFO:tasks.workunit.client.0.smithi092.stdout: create 262/39 file after the snapshot 2024-03-19T02:57:43.072 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/.snap/snap-subdir-test 2024-03-19T02:57:43.871 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/30 file after the snapshot 2024-03-19T02:57:43.872 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/31 file after the snapshot 2024-03-19T02:57:43.874 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/32 file after the snapshot 2024-03-19T02:57:43.875 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/33 file after the snapshot 2024-03-19T02:57:43.877 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/34 file after the snapshot 2024-03-19T02:57:43.879 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/35 file after the snapshot 2024-03-19T02:57:43.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/36 file after the snapshot 2024-03-19T02:57:43.881 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/37 file after the snapshot 2024-03-19T02:57:43.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/38 file after the snapshot 2024-03-19T02:57:43.884 INFO:tasks.workunit.client.0.smithi092.stdout: create 263/39 file after the snapshot 2024-03-19T02:57:43.886 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/.snap/snap-subdir-test 2024-03-19T02:57:43.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/30 file after the snapshot 2024-03-19T02:57:43.924 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/31 file after the snapshot 2024-03-19T02:57:43.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/32 file after the snapshot 2024-03-19T02:57:43.928 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/33 file after the snapshot 2024-03-19T02:57:43.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/34 file after the snapshot 2024-03-19T02:57:43.932 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/35 file after the snapshot 2024-03-19T02:57:43.934 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/36 file after the snapshot 2024-03-19T02:57:43.935 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/37 file after the snapshot 2024-03-19T02:57:43.937 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/38 file after the snapshot 2024-03-19T02:57:43.939 INFO:tasks.workunit.client.0.smithi092.stdout: create 264/39 file after the snapshot 2024-03-19T02:57:43.941 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/.snap/snap-subdir-test 2024-03-19T02:57:43.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/30 file after the snapshot 2024-03-19T02:57:43.977 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/31 file after the snapshot 2024-03-19T02:57:43.979 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/32 file after the snapshot 2024-03-19T02:57:43.981 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/33 file after the snapshot 2024-03-19T02:57:43.983 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/34 file after the snapshot 2024-03-19T02:57:43.985 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/35 file after the snapshot 2024-03-19T02:57:43.987 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/36 file after the snapshot 2024-03-19T02:57:43.989 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/37 file after the snapshot 2024-03-19T02:57:43.991 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/38 file after the snapshot 2024-03-19T02:57:43.993 INFO:tasks.workunit.client.0.smithi092.stdout: create 265/39 file after the snapshot 2024-03-19T02:57:43.994 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/.snap/snap-subdir-test 2024-03-19T02:57:44.028 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/30 file after the snapshot 2024-03-19T02:57:44.030 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/31 file after the snapshot 2024-03-19T02:57:44.032 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/32 file after the snapshot 2024-03-19T02:57:44.034 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/33 file after the snapshot 2024-03-19T02:57:44.036 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/34 file after the snapshot 2024-03-19T02:57:44.038 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/35 file after the snapshot 2024-03-19T02:57:44.040 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/36 file after the snapshot 2024-03-19T02:57:44.042 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/37 file after the snapshot 2024-03-19T02:57:44.044 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/38 file after the snapshot 2024-03-19T02:57:44.045 INFO:tasks.workunit.client.0.smithi092.stdout: create 266/39 file after the snapshot 2024-03-19T02:57:44.047 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/.snap/snap-subdir-test 2024-03-19T02:57:44.081 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/30 file after the snapshot 2024-03-19T02:57:44.082 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/31 file after the snapshot 2024-03-19T02:57:44.083 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/32 file after the snapshot 2024-03-19T02:57:44.085 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/33 file after the snapshot 2024-03-19T02:57:44.086 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/34 file after the snapshot 2024-03-19T02:57:44.088 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/35 file after the snapshot 2024-03-19T02:57:44.089 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/36 file after the snapshot 2024-03-19T02:57:44.090 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/37 file after the snapshot 2024-03-19T02:57:44.092 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/38 file after the snapshot 2024-03-19T02:57:44.093 INFO:tasks.workunit.client.0.smithi092.stdout: create 267/39 file after the snapshot 2024-03-19T02:57:44.094 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/.snap/snap-subdir-test 2024-03-19T02:57:44.126 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/30 file after the snapshot 2024-03-19T02:57:44.127 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/31 file after the snapshot 2024-03-19T02:57:44.129 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/32 file after the snapshot 2024-03-19T02:57:44.130 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/33 file after the snapshot 2024-03-19T02:57:44.132 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/34 file after the snapshot 2024-03-19T02:57:44.133 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/35 file after the snapshot 2024-03-19T02:57:44.135 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/36 file after the snapshot 2024-03-19T02:57:44.136 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/37 file after the snapshot 2024-03-19T02:57:44.138 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/38 file after the snapshot 2024-03-19T02:57:44.139 INFO:tasks.workunit.client.0.smithi092.stdout: create 268/39 file after the snapshot 2024-03-19T02:57:44.140 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/.snap/snap-subdir-test 2024-03-19T02:57:44.177 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/30 file after the snapshot 2024-03-19T02:57:44.179 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/31 file after the snapshot 2024-03-19T02:57:44.180 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/32 file after the snapshot 2024-03-19T02:57:44.181 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/33 file after the snapshot 2024-03-19T02:57:44.183 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/34 file after the snapshot 2024-03-19T02:57:44.185 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/35 file after the snapshot 2024-03-19T02:57:44.187 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/36 file after the snapshot 2024-03-19T02:57:44.189 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/37 file after the snapshot 2024-03-19T02:57:44.190 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/38 file after the snapshot 2024-03-19T02:57:44.192 INFO:tasks.workunit.client.0.smithi092.stdout: create 269/39 file after the snapshot 2024-03-19T02:57:44.194 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/.snap/snap-subdir-test 2024-03-19T02:57:44.228 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/30 file after the snapshot 2024-03-19T02:57:44.229 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/31 file after the snapshot 2024-03-19T02:57:44.230 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/32 file after the snapshot 2024-03-19T02:57:44.232 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/33 file after the snapshot 2024-03-19T02:57:44.233 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/34 file after the snapshot 2024-03-19T02:57:44.235 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/35 file after the snapshot 2024-03-19T02:57:44.236 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/36 file after the snapshot 2024-03-19T02:57:44.237 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/37 file after the snapshot 2024-03-19T02:57:44.239 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/38 file after the snapshot 2024-03-19T02:57:44.240 INFO:tasks.workunit.client.0.smithi092.stdout: create 270/39 file after the snapshot 2024-03-19T02:57:44.242 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/.snap/snap-subdir-test 2024-03-19T02:57:44.274 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/30 file after the snapshot 2024-03-19T02:57:44.275 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/31 file after the snapshot 2024-03-19T02:57:44.277 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/32 file after the snapshot 2024-03-19T02:57:44.278 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/33 file after the snapshot 2024-03-19T02:57:44.279 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/34 file after the snapshot 2024-03-19T02:57:44.281 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/35 file after the snapshot 2024-03-19T02:57:44.282 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/36 file after the snapshot 2024-03-19T02:57:44.284 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/37 file after the snapshot 2024-03-19T02:57:44.285 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/38 file after the snapshot 2024-03-19T02:57:44.286 INFO:tasks.workunit.client.0.smithi092.stdout: create 271/39 file after the snapshot 2024-03-19T02:57:44.288 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/.snap/snap-subdir-test 2024-03-19T02:57:44.318 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/30 file after the snapshot 2024-03-19T02:57:44.319 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/31 file after the snapshot 2024-03-19T02:57:44.320 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/32 file after the snapshot 2024-03-19T02:57:44.322 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/33 file after the snapshot 2024-03-19T02:57:44.323 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/34 file after the snapshot 2024-03-19T02:57:44.325 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/35 file after the snapshot 2024-03-19T02:57:44.326 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/36 file after the snapshot 2024-03-19T02:57:44.327 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/37 file after the snapshot 2024-03-19T02:57:44.329 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/38 file after the snapshot 2024-03-19T02:57:44.330 INFO:tasks.workunit.client.0.smithi092.stdout: create 272/39 file after the snapshot 2024-03-19T02:57:44.332 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/.snap/snap-subdir-test 2024-03-19T02:57:44.364 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/30 file after the snapshot 2024-03-19T02:57:44.365 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/31 file after the snapshot 2024-03-19T02:57:44.367 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/32 file after the snapshot 2024-03-19T02:57:44.368 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/33 file after the snapshot 2024-03-19T02:57:44.369 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/34 file after the snapshot 2024-03-19T02:57:44.371 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/35 file after the snapshot 2024-03-19T02:57:44.372 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/36 file after the snapshot 2024-03-19T02:57:44.374 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/37 file after the snapshot 2024-03-19T02:57:44.375 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/38 file after the snapshot 2024-03-19T02:57:44.377 INFO:tasks.workunit.client.0.smithi092.stdout: create 273/39 file after the snapshot 2024-03-19T02:57:44.378 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/.snap/snap-subdir-test 2024-03-19T02:57:44.410 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/30 file after the snapshot 2024-03-19T02:57:44.411 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/31 file after the snapshot 2024-03-19T02:57:44.414 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/32 file after the snapshot 2024-03-19T02:57:44.414 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/33 file after the snapshot 2024-03-19T02:57:44.680 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/34 file after the snapshot 2024-03-19T02:57:44.682 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/35 file after the snapshot 2024-03-19T02:57:44.684 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/36 file after the snapshot 2024-03-19T02:57:44.686 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/37 file after the snapshot 2024-03-19T02:57:44.687 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/38 file after the snapshot 2024-03-19T02:57:44.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 274/39 file after the snapshot 2024-03-19T02:57:44.691 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/.snap/snap-subdir-test 2024-03-19T02:57:44.725 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/30 file after the snapshot 2024-03-19T02:57:44.726 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/31 file after the snapshot 2024-03-19T02:57:44.727 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/32 file after the snapshot 2024-03-19T02:57:44.729 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/33 file after the snapshot 2024-03-19T02:57:44.730 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/34 file after the snapshot 2024-03-19T02:57:44.732 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/35 file after the snapshot 2024-03-19T02:57:44.733 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/36 file after the snapshot 2024-03-19T02:57:44.734 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/37 file after the snapshot 2024-03-19T02:57:44.736 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/38 file after the snapshot 2024-03-19T02:57:44.737 INFO:tasks.workunit.client.0.smithi092.stdout: create 275/39 file after the snapshot 2024-03-19T02:57:44.739 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/.snap/snap-subdir-test 2024-03-19T02:57:44.772 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/30 file after the snapshot 2024-03-19T02:57:44.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/31 file after the snapshot 2024-03-19T02:57:44.776 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/32 file after the snapshot 2024-03-19T02:57:44.778 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/33 file after the snapshot 2024-03-19T02:57:44.780 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/34 file after the snapshot 2024-03-19T02:57:44.782 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/35 file after the snapshot 2024-03-19T02:57:44.783 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/36 file after the snapshot 2024-03-19T02:57:44.785 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/37 file after the snapshot 2024-03-19T02:57:44.787 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/38 file after the snapshot 2024-03-19T02:57:44.789 INFO:tasks.workunit.client.0.smithi092.stdout: create 276/39 file after the snapshot 2024-03-19T02:57:44.791 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/.snap/snap-subdir-test 2024-03-19T02:57:44.827 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/30 file after the snapshot 2024-03-19T02:57:44.829 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/31 file after the snapshot 2024-03-19T02:57:44.830 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/32 file after the snapshot 2024-03-19T02:57:44.831 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/33 file after the snapshot 2024-03-19T02:57:44.832 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/34 file after the snapshot 2024-03-19T02:57:44.833 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/35 file after the snapshot 2024-03-19T02:57:44.835 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/36 file after the snapshot 2024-03-19T02:57:44.836 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/37 file after the snapshot 2024-03-19T02:57:44.837 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/38 file after the snapshot 2024-03-19T02:57:44.839 INFO:tasks.workunit.client.0.smithi092.stdout: create 277/39 file after the snapshot 2024-03-19T02:57:44.840 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/.snap/snap-subdir-test 2024-03-19T02:57:44.872 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/30 file after the snapshot 2024-03-19T02:57:44.873 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/31 file after the snapshot 2024-03-19T02:57:44.875 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/32 file after the snapshot 2024-03-19T02:57:44.876 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/33 file after the snapshot 2024-03-19T02:57:44.878 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/34 file after the snapshot 2024-03-19T02:57:44.879 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/35 file after the snapshot 2024-03-19T02:57:44.880 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/36 file after the snapshot 2024-03-19T02:57:44.882 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/37 file after the snapshot 2024-03-19T02:57:44.883 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/38 file after the snapshot 2024-03-19T02:57:44.885 INFO:tasks.workunit.client.0.smithi092.stdout: create 278/39 file after the snapshot 2024-03-19T02:57:44.886 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/.snap/snap-subdir-test 2024-03-19T02:57:44.919 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/30 file after the snapshot 2024-03-19T02:57:44.920 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/31 file after the snapshot 2024-03-19T02:57:44.922 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/32 file after the snapshot 2024-03-19T02:57:44.923 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/33 file after the snapshot 2024-03-19T02:57:44.925 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/34 file after the snapshot 2024-03-19T02:57:44.926 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/35 file after the snapshot 2024-03-19T02:57:44.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/36 file after the snapshot 2024-03-19T02:57:44.930 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/37 file after the snapshot 2024-03-19T02:57:44.931 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/38 file after the snapshot 2024-03-19T02:57:44.931 INFO:tasks.workunit.client.0.smithi092.stdout: create 279/39 file after the snapshot 2024-03-19T02:57:44.933 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/.snap/snap-subdir-test 2024-03-19T02:57:44.965 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/30 file after the snapshot 2024-03-19T02:57:44.966 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/31 file after the snapshot 2024-03-19T02:57:44.967 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/32 file after the snapshot 2024-03-19T02:57:44.969 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/33 file after the snapshot 2024-03-19T02:57:44.970 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/34 file after the snapshot 2024-03-19T02:57:44.972 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/35 file after the snapshot 2024-03-19T02:57:44.973 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/36 file after the snapshot 2024-03-19T02:57:44.975 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/37 file after the snapshot 2024-03-19T02:57:44.976 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/38 file after the snapshot 2024-03-19T02:57:44.977 INFO:tasks.workunit.client.0.smithi092.stdout: create 280/39 file after the snapshot 2024-03-19T02:57:44.979 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/.snap/snap-subdir-test 2024-03-19T02:57:45.010 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/30 file after the snapshot 2024-03-19T02:57:45.012 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/31 file after the snapshot 2024-03-19T02:57:45.014 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/32 file after the snapshot 2024-03-19T02:57:45.015 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/33 file after the snapshot 2024-03-19T02:57:45.016 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/34 file after the snapshot 2024-03-19T02:57:45.018 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/35 file after the snapshot 2024-03-19T02:57:45.019 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/36 file after the snapshot 2024-03-19T02:57:45.021 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/37 file after the snapshot 2024-03-19T02:57:45.023 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/38 file after the snapshot 2024-03-19T02:57:45.024 INFO:tasks.workunit.client.0.smithi092.stdout: create 281/39 file after the snapshot 2024-03-19T02:57:45.026 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/.snap/snap-subdir-test 2024-03-19T02:57:45.058 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/30 file after the snapshot 2024-03-19T02:57:45.060 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/31 file after the snapshot 2024-03-19T02:57:45.062 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/32 file after the snapshot 2024-03-19T02:57:45.123 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/33 file after the snapshot 2024-03-19T02:57:45.124 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/34 file after the snapshot 2024-03-19T02:57:45.126 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/35 file after the snapshot 2024-03-19T02:57:45.127 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/36 file after the snapshot 2024-03-19T02:57:45.129 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/37 file after the snapshot 2024-03-19T02:57:45.392 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/38 file after the snapshot 2024-03-19T02:57:45.393 INFO:tasks.workunit.client.0.smithi092.stdout: create 282/39 file after the snapshot 2024-03-19T02:57:45.395 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/.snap/snap-subdir-test 2024-03-19T02:57:45.425 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/30 file after the snapshot 2024-03-19T02:57:45.430 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/31 file after the snapshot 2024-03-19T02:57:45.431 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/32 file after the snapshot 2024-03-19T02:57:45.432 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/33 file after the snapshot 2024-03-19T02:57:45.434 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/34 file after the snapshot 2024-03-19T02:57:45.436 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/35 file after the snapshot 2024-03-19T02:57:45.437 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/36 file after the snapshot 2024-03-19T02:57:45.439 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/37 file after the snapshot 2024-03-19T02:57:45.442 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/38 file after the snapshot 2024-03-19T02:57:45.444 INFO:tasks.workunit.client.0.smithi092.stdout: create 283/39 file after the snapshot 2024-03-19T02:57:45.445 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/.snap/snap-subdir-test 2024-03-19T02:57:45.481 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/30 file after the snapshot 2024-03-19T02:57:45.483 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/31 file after the snapshot 2024-03-19T02:57:45.485 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/32 file after the snapshot 2024-03-19T02:57:45.487 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/33 file after the snapshot 2024-03-19T02:57:45.489 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/34 file after the snapshot 2024-03-19T02:57:45.491 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/35 file after the snapshot 2024-03-19T02:57:45.493 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/36 file after the snapshot 2024-03-19T02:57:45.495 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/37 file after the snapshot 2024-03-19T02:57:45.497 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/38 file after the snapshot 2024-03-19T02:57:45.499 INFO:tasks.workunit.client.0.smithi092.stdout: create 284/39 file after the snapshot 2024-03-19T02:57:45.500 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/.snap/snap-subdir-test 2024-03-19T02:57:45.532 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/30 file after the snapshot 2024-03-19T02:57:45.534 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/31 file after the snapshot 2024-03-19T02:57:45.536 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/32 file after the snapshot 2024-03-19T02:57:45.538 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/33 file after the snapshot 2024-03-19T02:57:45.540 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/34 file after the snapshot 2024-03-19T02:57:45.542 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/35 file after the snapshot 2024-03-19T02:57:45.544 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/36 file after the snapshot 2024-03-19T02:57:45.546 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/37 file after the snapshot 2024-03-19T02:57:45.548 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/38 file after the snapshot 2024-03-19T02:57:45.550 INFO:tasks.workunit.client.0.smithi092.stdout: create 285/39 file after the snapshot 2024-03-19T02:57:45.552 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/.snap/snap-subdir-test 2024-03-19T02:57:45.584 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/30 file after the snapshot 2024-03-19T02:57:45.586 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/31 file after the snapshot 2024-03-19T02:57:45.588 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/32 file after the snapshot 2024-03-19T02:57:45.590 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/33 file after the snapshot 2024-03-19T02:57:45.592 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/34 file after the snapshot 2024-03-19T02:57:45.594 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/35 file after the snapshot 2024-03-19T02:57:45.596 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/36 file after the snapshot 2024-03-19T02:57:45.598 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/37 file after the snapshot 2024-03-19T02:57:45.600 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/38 file after the snapshot 2024-03-19T02:57:45.602 INFO:tasks.workunit.client.0.smithi092.stdout: create 286/39 file after the snapshot 2024-03-19T02:57:45.604 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/.snap/snap-subdir-test 2024-03-19T02:57:45.637 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/30 file after the snapshot 2024-03-19T02:57:45.638 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/31 file after the snapshot 2024-03-19T02:57:45.640 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/32 file after the snapshot 2024-03-19T02:57:45.641 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/33 file after the snapshot 2024-03-19T02:57:45.642 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/34 file after the snapshot 2024-03-19T02:57:45.643 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/35 file after the snapshot 2024-03-19T02:57:45.645 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/36 file after the snapshot 2024-03-19T02:57:45.646 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/37 file after the snapshot 2024-03-19T02:57:45.648 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/38 file after the snapshot 2024-03-19T02:57:45.649 INFO:tasks.workunit.client.0.smithi092.stdout: create 287/39 file after the snapshot 2024-03-19T02:57:45.650 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/.snap/snap-subdir-test 2024-03-19T02:57:45.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/30 file after the snapshot 2024-03-19T02:57:45.685 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/31 file after the snapshot 2024-03-19T02:57:45.687 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/32 file after the snapshot 2024-03-19T02:57:45.689 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/33 file after the snapshot 2024-03-19T02:57:45.691 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/34 file after the snapshot 2024-03-19T02:57:45.693 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/35 file after the snapshot 2024-03-19T02:57:45.695 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/36 file after the snapshot 2024-03-19T02:57:45.697 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/37 file after the snapshot 2024-03-19T02:57:45.699 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/38 file after the snapshot 2024-03-19T02:57:45.700 INFO:tasks.workunit.client.0.smithi092.stdout: create 288/39 file after the snapshot 2024-03-19T02:57:45.701 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/.snap/snap-subdir-test 2024-03-19T02:57:45.736 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/30 file after the snapshot 2024-03-19T02:57:45.739 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/31 file after the snapshot 2024-03-19T02:57:45.741 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/32 file after the snapshot 2024-03-19T02:57:45.743 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/33 file after the snapshot 2024-03-19T02:57:45.745 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/34 file after the snapshot 2024-03-19T02:57:45.746 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/35 file after the snapshot 2024-03-19T02:57:45.748 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/36 file after the snapshot 2024-03-19T02:57:45.750 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/37 file after the snapshot 2024-03-19T02:57:45.752 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/38 file after the snapshot 2024-03-19T02:57:45.754 INFO:tasks.workunit.client.0.smithi092.stdout: create 289/39 file after the snapshot 2024-03-19T02:57:45.756 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/.snap/snap-subdir-test 2024-03-19T02:57:45.793 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/30 file after the snapshot 2024-03-19T02:57:45.795 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/31 file after the snapshot 2024-03-19T02:57:45.796 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/32 file after the snapshot 2024-03-19T02:57:45.798 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/33 file after the snapshot 2024-03-19T02:57:45.800 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/34 file after the snapshot 2024-03-19T02:57:45.802 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/35 file after the snapshot 2024-03-19T02:57:45.804 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/36 file after the snapshot 2024-03-19T02:57:45.806 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/37 file after the snapshot 2024-03-19T02:57:45.808 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/38 file after the snapshot 2024-03-19T02:57:45.810 INFO:tasks.workunit.client.0.smithi092.stdout: create 290/39 file after the snapshot 2024-03-19T02:57:45.812 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/.snap/snap-subdir-test 2024-03-19T02:57:45.849 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/30 file after the snapshot 2024-03-19T02:57:45.849 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/31 file after the snapshot 2024-03-19T02:57:45.851 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/32 file after the snapshot 2024-03-19T02:57:45.852 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/33 file after the snapshot 2024-03-19T02:57:45.854 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/34 file after the snapshot 2024-03-19T02:57:45.855 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/35 file after the snapshot 2024-03-19T02:57:45.857 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/36 file after the snapshot 2024-03-19T02:57:45.858 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/37 file after the snapshot 2024-03-19T02:57:45.859 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/38 file after the snapshot 2024-03-19T02:57:45.861 INFO:tasks.workunit.client.0.smithi092.stdout: create 291/39 file after the snapshot 2024-03-19T02:57:45.862 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/.snap/snap-subdir-test 2024-03-19T02:57:46.516 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/30 file after the snapshot 2024-03-19T02:57:46.517 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/31 file after the snapshot 2024-03-19T02:57:46.519 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/32 file after the snapshot 2024-03-19T02:57:46.520 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/33 file after the snapshot 2024-03-19T02:57:46.521 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/34 file after the snapshot 2024-03-19T02:57:46.523 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/35 file after the snapshot 2024-03-19T02:57:46.524 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/36 file after the snapshot 2024-03-19T02:57:46.526 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/37 file after the snapshot 2024-03-19T02:57:46.528 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/38 file after the snapshot 2024-03-19T02:57:46.530 INFO:tasks.workunit.client.0.smithi092.stdout: create 292/39 file after the snapshot 2024-03-19T02:57:46.531 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/.snap/snap-subdir-test 2024-03-19T02:57:46.563 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/30 file after the snapshot 2024-03-19T02:57:46.565 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/31 file after the snapshot 2024-03-19T02:57:46.567 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/32 file after the snapshot 2024-03-19T02:57:46.569 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/33 file after the snapshot 2024-03-19T02:57:46.571 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/34 file after the snapshot 2024-03-19T02:57:46.573 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/35 file after the snapshot 2024-03-19T02:57:46.575 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/36 file after the snapshot 2024-03-19T02:57:46.577 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/37 file after the snapshot 2024-03-19T02:57:46.579 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/38 file after the snapshot 2024-03-19T02:57:46.581 INFO:tasks.workunit.client.0.smithi092.stdout: create 293/39 file after the snapshot 2024-03-19T02:57:46.583 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/.snap/snap-subdir-test 2024-03-19T02:57:46.615 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/30 file after the snapshot 2024-03-19T02:57:46.617 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/31 file after the snapshot 2024-03-19T02:57:46.619 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/32 file after the snapshot 2024-03-19T02:57:46.621 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/33 file after the snapshot 2024-03-19T02:57:46.623 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/34 file after the snapshot 2024-03-19T02:57:46.625 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/35 file after the snapshot 2024-03-19T02:57:46.627 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/36 file after the snapshot 2024-03-19T02:57:46.629 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/37 file after the snapshot 2024-03-19T02:57:46.631 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/38 file after the snapshot 2024-03-19T02:57:46.633 INFO:tasks.workunit.client.0.smithi092.stdout: create 294/39 file after the snapshot 2024-03-19T02:57:46.635 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/.snap/snap-subdir-test 2024-03-19T02:57:46.668 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/30 file after the snapshot 2024-03-19T02:57:46.670 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/31 file after the snapshot 2024-03-19T02:57:46.671 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/32 file after the snapshot 2024-03-19T02:57:46.673 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/33 file after the snapshot 2024-03-19T02:57:46.674 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/34 file after the snapshot 2024-03-19T02:57:46.676 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/35 file after the snapshot 2024-03-19T02:57:46.677 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/36 file after the snapshot 2024-03-19T02:57:46.678 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/37 file after the snapshot 2024-03-19T02:57:46.680 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/38 file after the snapshot 2024-03-19T02:57:46.681 INFO:tasks.workunit.client.0.smithi092.stdout: create 295/39 file after the snapshot 2024-03-19T02:57:46.683 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/.snap/snap-subdir-test 2024-03-19T02:57:46.716 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/30 file after the snapshot 2024-03-19T02:57:46.718 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/31 file after the snapshot 2024-03-19T02:57:46.719 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/32 file after the snapshot 2024-03-19T02:57:46.721 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/33 file after the snapshot 2024-03-19T02:57:46.722 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/34 file after the snapshot 2024-03-19T02:57:46.724 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/35 file after the snapshot 2024-03-19T02:57:46.725 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/36 file after the snapshot 2024-03-19T02:57:46.727 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/37 file after the snapshot 2024-03-19T02:57:46.728 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/38 file after the snapshot 2024-03-19T02:57:46.730 INFO:tasks.workunit.client.0.smithi092.stdout: create 296/39 file after the snapshot 2024-03-19T02:57:46.731 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/.snap/snap-subdir-test 2024-03-19T02:57:46.759 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/30 file after the snapshot 2024-03-19T02:57:46.761 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/31 file after the snapshot 2024-03-19T02:57:46.762 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/32 file after the snapshot 2024-03-19T02:57:46.764 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/33 file after the snapshot 2024-03-19T02:57:46.765 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/34 file after the snapshot 2024-03-19T02:57:46.767 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/35 file after the snapshot 2024-03-19T02:57:46.768 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/36 file after the snapshot 2024-03-19T02:57:46.770 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/37 file after the snapshot 2024-03-19T02:57:46.771 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/38 file after the snapshot 2024-03-19T02:57:46.772 INFO:tasks.workunit.client.0.smithi092.stdout: create 297/39 file after the snapshot 2024-03-19T02:57:46.774 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/.snap/snap-subdir-test 2024-03-19T02:57:46.802 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/30 file after the snapshot 2024-03-19T02:57:46.863 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/31 file after the snapshot 2024-03-19T02:57:47.263 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/32 file after the snapshot 2024-03-19T02:57:47.264 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/33 file after the snapshot 2024-03-19T02:57:47.266 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/34 file after the snapshot 2024-03-19T02:57:47.267 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/35 file after the snapshot 2024-03-19T02:57:47.269 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/36 file after the snapshot 2024-03-19T02:57:47.270 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/37 file after the snapshot 2024-03-19T02:57:47.272 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/38 file after the snapshot 2024-03-19T02:57:47.273 INFO:tasks.workunit.client.0.smithi092.stdout: create 298/39 file after the snapshot 2024-03-19T02:57:47.275 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/.snap/snap-subdir-test 2024-03-19T02:57:47.320 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/30 file after the snapshot 2024-03-19T02:57:47.321 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/31 file after the snapshot 2024-03-19T02:57:47.323 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/32 file after the snapshot 2024-03-19T02:57:47.324 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/33 file after the snapshot 2024-03-19T02:57:47.325 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/34 file after the snapshot 2024-03-19T02:57:47.327 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/35 file after the snapshot 2024-03-19T02:57:47.329 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/36 file after the snapshot 2024-03-19T02:57:47.330 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/37 file after the snapshot 2024-03-19T02:57:47.332 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/38 file after the snapshot 2024-03-19T02:57:47.333 INFO:tasks.workunit.client.0.smithi092.stdout: create 299/39 file after the snapshot 2024-03-19T02:57:47.335 INFO:tasks.workunit.client.0.smithi092.stdout:Take second snapshot .snap/test2 2024-03-19T02:57:47.919 INFO:tasks.workunit.client.0.smithi092.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-03-19T02:57:48.720 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-03-19T02:57:48.721 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-03-19T02:57:48.722 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-03-19T02:57:48.723 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-03-19T02:57:48.724 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-03-19T02:57:48.725 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-03-19T02:57:48.726 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-03-19T02:57:48.727 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-03-19T02:57:48.728 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-03-19T02:57:48.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-03-19T02:57:48.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-03-19T02:57:48.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-03-19T02:57:48.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-03-19T02:57:49.474 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-03-19T02:57:49.474 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-03-19T02:57:49.475 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-03-19T02:57:49.476 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-03-19T02:57:49.477 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-03-19T02:57:49.478 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-03-19T02:57:49.479 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-03-19T02:57:49.480 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-03-19T02:57:52.776 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-03-19T02:57:52.776 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-03-19T02:57:52.776 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-03-19T02:57:52.776 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-03-19T02:57:52.776 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-03-19T02:57:52.777 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-03-19T02:57:52.778 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-03-19T02:57:52.779 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-03-19T02:57:52.780 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-03-19T02:57:52.781 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-03-19T02:57:52.782 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-03-19T02:57:53.013 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-03-19T02:57:53.014 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-03-19T02:57:53.015 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-03-19T02:57:53.016 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-03-19T02:57:53.017 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-03-19T02:57:53.018 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-03-19T02:57:53.019 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-03-19T02:57:53.020 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-03-19T02:57:55.090 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-03-19T02:57:55.091 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-03-19T02:57:55.092 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-03-19T02:57:55.093 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-03-19T02:57:55.094 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-03-19T02:57:55.095 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-03-19T02:57:55.096 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-03-19T02:57:55.759 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-03-19T02:57:55.759 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-03-19T02:57:55.759 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-03-19T02:57:55.759 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-03-19T02:57:55.759 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-03-19T02:57:55.760 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-03-19T02:57:55.761 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-03-19T02:57:55.762 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-03-19T02:57:55.763 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-03-19T02:57:55.764 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-03-19T02:57:55.765 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-03-19T02:57:55.766 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-03-19T02:57:55.766 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-03-19T02:57:55.766 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-03-19T02:57:55.766 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-03-19T02:57:55.766 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-03-19T02:57:55.767 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-03-19T02:57:55.768 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-03-19T02:57:55.769 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-03-19T02:57:55.770 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-03-19T02:57:55.770 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-03-19T02:57:55.770 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-03-19T02:57:55.770 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-03-19T02:57:55.770 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-03-19T02:57:56.414 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-03-19T02:57:56.414 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-03-19T02:57:56.414 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-03-19T02:57:56.414 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-03-19T02:57:56.414 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-03-19T02:57:56.415 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-03-19T02:57:56.416 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-03-19T02:57:56.417 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-03-19T02:57:56.418 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-03-19T02:57:56.419 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-03-19T02:57:56.420 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-03-19T02:57:57.060 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-03-19T02:57:57.060 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-03-19T02:57:57.060 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-03-19T02:57:57.061 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-03-19T02:57:57.062 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-03-19T02:57:57.063 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-03-19T02:57:57.064 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-03-19T02:57:57.065 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-03-19T02:57:57.066 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-03-19T02:57:57.729 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-03-19T02:57:57.730 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-03-19T02:57:57.731 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-03-19T02:57:57.732 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-03-19T02:57:57.733 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-03-19T02:57:57.734 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-03-19T02:57:57.735 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-03-19T02:57:57.736 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-03-19T02:57:57.957 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-03-19T02:57:57.958 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-03-19T02:57:57.959 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-03-19T02:57:57.959 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-03-19T02:57:57.959 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-03-19T02:57:57.959 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-03-19T02:57:57.959 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-03-19T02:57:57.960 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-03-19T02:57:57.960 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-03-19T02:57:57.960 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-03-19T02:57:57.961 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-03-19T02:57:57.962 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-03-19T02:57:57.963 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-03-19T02:57:57.964 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-03-19T02:57:57.965 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-03-19T02:57:57.965 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-03-19T02:57:57.965 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-03-19T02:57:57.965 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-03-19T02:57:57.965 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-03-19T02:57:58.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-03-19T02:57:58.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-03-19T02:57:58.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-03-19T02:57:58.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-03-19T02:57:58.835 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-03-19T02:57:58.836 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-03-19T02:57:58.837 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-03-19T02:57:59.143 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-03-19T02:57:59.144 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-03-19T02:57:59.145 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-03-19T02:57:59.146 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-03-19T02:57:59.147 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-03-19T02:57:59.150 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-03-19T02:57:59.151 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-03-19T02:57:59.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:57:59.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:58:00.710 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-03-19T02:58:00.711 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-03-19T02:58:00.712 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-03-19T02:58:00.713 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-03-19T02:58:00.714 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-03-19T02:58:00.715 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-03-19T02:58:00.716 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-03-19T02:58:02.828 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-03-19T02:58:02.829 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-03-19T02:58:02.830 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-03-19T02:58:02.831 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-03-19T02:58:02.832 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-03-19T02:58:02.833 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-03-19T02:58:02.834 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-03-19T02:58:02.838 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-03-19T02:58:03.390 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-03-19T02:58:03.391 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-03-19T02:58:03.392 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-03-19T02:58:03.393 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-03-19T02:58:03.394 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-03-19T02:58:03.395 INFO:tasks.workunit.client.0.smithi092.stdout:Take third snapshot .snap/test3 2024-03-19T02:58:03.511 INFO:tasks.workunit.client.0.smithi092.stdout:Delete the snapshots... 2024-03-19T02:58:06.688 INFO:tasks.workunit.client.0.smithi092.stdout:.//.snap/test1 2024-03-19T02:58:06.689 INFO:tasks.workunit.client.0.smithi092.stdout:.//.snap/test2 2024-03-19T02:58:06.689 INFO:tasks.workunit.client.0.smithi092.stdout:.//.snap/test3 2024-03-19T02:58:06.697 INFO:tasks.workunit.client.0.smithi092.stdout:./274/.snap/snap-subdir-test 2024-03-19T02:58:06.707 INFO:tasks.workunit.client.0.smithi092.stdout:./166/.snap/snap-subdir-test 2024-03-19T02:58:06.715 INFO:tasks.workunit.client.0.smithi092.stdout:./256/.snap/snap-subdir-test 2024-03-19T02:58:06.722 INFO:tasks.workunit.client.0.smithi092.stdout:./247/.snap/snap-subdir-test 2024-03-19T02:58:06.730 INFO:tasks.workunit.client.0.smithi092.stdout:./248/.snap/snap-subdir-test 2024-03-19T02:58:06.739 INFO:tasks.workunit.client.0.smithi092.stdout:./147/.snap/snap-subdir-test 2024-03-19T02:58:06.748 INFO:tasks.workunit.client.0.smithi092.stdout:./151/.snap/snap-subdir-test 2024-03-19T02:58:06.756 INFO:tasks.workunit.client.0.smithi092.stdout:./170/.snap/snap-subdir-test 2024-03-19T02:58:06.766 INFO:tasks.workunit.client.0.smithi092.stdout:./167/.snap/snap-subdir-test 2024-03-19T02:58:06.818 INFO:tasks.workunit.client.0.smithi092.stdout:./140/.snap/snap-subdir-test 2024-03-19T02:58:09.299 INFO:tasks.workunit.client.0.smithi092.stdout:./142/.snap/snap-subdir-test 2024-03-19T02:58:09.872 INFO:tasks.workunit.client.0.smithi092.stdout:./150/.snap/snap-subdir-test 2024-03-19T02:58:09.881 INFO:tasks.workunit.client.0.smithi092.stdout:./273/.snap/snap-subdir-test 2024-03-19T02:58:09.890 INFO:tasks.workunit.client.0.smithi092.stdout:./269/.snap/snap-subdir-test 2024-03-19T02:58:09.898 INFO:tasks.workunit.client.0.smithi092.stdout:./105/.snap/snap-subdir-test 2024-03-19T02:58:09.906 INFO:tasks.workunit.client.0.smithi092.stdout:./210/.snap/snap-subdir-test 2024-03-19T02:58:09.914 INFO:tasks.workunit.client.0.smithi092.stdout:./265/.snap/snap-subdir-test 2024-03-19T02:58:10.383 INFO:tasks.workunit.client.0.smithi092.stdout:./218/.snap/snap-subdir-test 2024-03-19T02:58:10.393 INFO:tasks.workunit.client.0.smithi092.stdout:./290/.snap/snap-subdir-test 2024-03-19T02:58:10.406 INFO:tasks.workunit.client.0.smithi092.stdout:./149/.snap/snap-subdir-test 2024-03-19T02:58:10.413 INFO:tasks.workunit.client.0.smithi092.stdout:./109/.snap/snap-subdir-test 2024-03-19T02:58:10.421 INFO:tasks.workunit.client.0.smithi092.stdout:./292/.snap/snap-subdir-test 2024-03-19T02:58:10.429 INFO:tasks.workunit.client.0.smithi092.stdout:./242/.snap/snap-subdir-test 2024-03-19T02:58:10.438 INFO:tasks.workunit.client.0.smithi092.stdout:./158/.snap/snap-subdir-test 2024-03-19T02:58:10.447 INFO:tasks.workunit.client.0.smithi092.stdout:./243/.snap/snap-subdir-test 2024-03-19T02:58:10.457 INFO:tasks.workunit.client.0.smithi092.stdout:./106/.snap/snap-subdir-test 2024-03-19T02:58:10.466 INFO:tasks.workunit.client.0.smithi092.stdout:./250/.snap/snap-subdir-test 2024-03-19T02:58:10.476 INFO:tasks.workunit.client.0.smithi092.stdout:./221/.snap/snap-subdir-test 2024-03-19T02:58:10.484 INFO:tasks.workunit.client.0.smithi092.stdout:./244/.snap/snap-subdir-test 2024-03-19T02:58:10.493 INFO:tasks.workunit.client.0.smithi092.stdout:./258/.snap/snap-subdir-test 2024-03-19T02:58:10.503 INFO:tasks.workunit.client.0.smithi092.stdout:./268/.snap/snap-subdir-test 2024-03-19T02:58:10.512 INFO:tasks.workunit.client.0.smithi092.stdout:./189/.snap/snap-subdir-test 2024-03-19T02:58:10.521 INFO:tasks.workunit.client.0.smithi092.stdout:./285/.snap/snap-subdir-test 2024-03-19T02:58:10.530 INFO:tasks.workunit.client.0.smithi092.stdout:./208/.snap/snap-subdir-test 2024-03-19T02:58:10.539 INFO:tasks.workunit.client.0.smithi092.stdout:./288/.snap/snap-subdir-test 2024-03-19T02:58:10.549 INFO:tasks.workunit.client.0.smithi092.stdout:./209/.snap/snap-subdir-test 2024-03-19T02:58:10.559 INFO:tasks.workunit.client.0.smithi092.stdout:./253/.snap/snap-subdir-test 2024-03-19T02:58:10.568 INFO:tasks.workunit.client.0.smithi092.stdout:./249/.snap/snap-subdir-test 2024-03-19T02:58:10.579 INFO:tasks.workunit.client.0.smithi092.stdout:./133/.snap/snap-subdir-test 2024-03-19T02:58:10.588 INFO:tasks.workunit.client.0.smithi092.stdout:./291/.snap/snap-subdir-test 2024-03-19T02:58:10.598 INFO:tasks.workunit.client.0.smithi092.stdout:./226/.snap/snap-subdir-test 2024-03-19T02:58:10.953 INFO:tasks.workunit.client.0.smithi092.stdout:./173/.snap/snap-subdir-test 2024-03-19T02:58:11.278 INFO:tasks.workunit.client.0.smithi092.stdout:./174/.snap/snap-subdir-test 2024-03-19T02:58:11.288 INFO:tasks.workunit.client.0.smithi092.stdout:./299/.snap/snap-subdir-test 2024-03-19T02:58:11.297 INFO:tasks.workunit.client.0.smithi092.stdout:./262/.snap/snap-subdir-test 2024-03-19T02:58:11.305 INFO:tasks.workunit.client.0.smithi092.stdout:./231/.snap/snap-subdir-test 2024-03-19T02:58:11.313 INFO:tasks.workunit.client.0.smithi092.stdout:./214/.snap/snap-subdir-test 2024-03-19T02:58:11.322 INFO:tasks.workunit.client.0.smithi092.stdout:./281/.snap/snap-subdir-test 2024-03-19T02:58:11.330 INFO:tasks.workunit.client.0.smithi092.stdout:./289/.snap/snap-subdir-test 2024-03-19T02:58:11.338 INFO:tasks.workunit.client.0.smithi092.stdout:./283/.snap/snap-subdir-test 2024-03-19T02:58:11.346 INFO:tasks.workunit.client.0.smithi092.stdout:./232/.snap/snap-subdir-test 2024-03-19T02:58:11.619 INFO:tasks.workunit.client.0.smithi092.stdout:./298/.snap/snap-subdir-test 2024-03-19T02:58:11.629 INFO:tasks.workunit.client.0.smithi092.stdout:./233/.snap/snap-subdir-test 2024-03-19T02:58:11.638 INFO:tasks.workunit.client.0.smithi092.stdout:./155/.snap/snap-subdir-test 2024-03-19T02:58:11.647 INFO:tasks.workunit.client.0.smithi092.stdout:./114/.snap/snap-subdir-test 2024-03-19T02:58:11.657 INFO:tasks.workunit.client.0.smithi092.stdout:./159/.snap/snap-subdir-test 2024-03-19T02:58:11.668 INFO:tasks.workunit.client.0.smithi092.stdout:./187/.snap/snap-subdir-test 2024-03-19T02:58:11.679 INFO:tasks.workunit.client.0.smithi092.stdout:./185/.snap/snap-subdir-test 2024-03-19T02:58:11.688 INFO:tasks.workunit.client.0.smithi092.stdout:./293/.snap/snap-subdir-test 2024-03-19T02:58:11.697 INFO:tasks.workunit.client.0.smithi092.stdout:./228/.snap/snap-subdir-test 2024-03-19T02:58:11.705 INFO:tasks.workunit.client.0.smithi092.stdout:./120/.snap/snap-subdir-test 2024-03-19T02:58:11.713 INFO:tasks.workunit.client.0.smithi092.stdout:./110/.snap/snap-subdir-test 2024-03-19T02:58:11.721 INFO:tasks.workunit.client.0.smithi092.stdout:./267/.snap/snap-subdir-test 2024-03-19T02:58:11.729 INFO:tasks.workunit.client.0.smithi092.stdout:./202/.snap/snap-subdir-test 2024-03-19T02:58:11.737 INFO:tasks.workunit.client.0.smithi092.stdout:./137/.snap/snap-subdir-test 2024-03-19T02:58:11.745 INFO:tasks.workunit.client.0.smithi092.stdout:./246/.snap/snap-subdir-test 2024-03-19T02:58:11.755 INFO:tasks.workunit.client.0.smithi092.stdout:./219/.snap/snap-subdir-test 2024-03-19T02:58:11.763 INFO:tasks.workunit.client.0.smithi092.stdout:./198/.snap/snap-subdir-test 2024-03-19T02:58:11.771 INFO:tasks.workunit.client.0.smithi092.stdout:./136/.snap/snap-subdir-test 2024-03-19T02:58:13.417 INFO:tasks.workunit.client.0.smithi092.stdout:./270/.snap/snap-subdir-test 2024-03-19T02:58:13.428 INFO:tasks.workunit.client.0.smithi092.stdout:./125/.snap/snap-subdir-test 2024-03-19T02:58:13.436 INFO:tasks.workunit.client.0.smithi092.stdout:./194/.snap/snap-subdir-test 2024-03-19T02:58:13.445 INFO:tasks.workunit.client.0.smithi092.stdout:./119/.snap/snap-subdir-test 2024-03-19T02:58:13.452 INFO:tasks.workunit.client.0.smithi092.stdout:./297/.snap/snap-subdir-test 2024-03-19T02:58:13.461 INFO:tasks.workunit.client.0.smithi092.stdout:./230/.snap/snap-subdir-test 2024-03-19T02:58:13.469 INFO:tasks.workunit.client.0.smithi092.stdout:./260/.snap/snap-subdir-test 2024-03-19T02:58:13.479 INFO:tasks.workunit.client.0.smithi092.stdout:./157/.snap/snap-subdir-test 2024-03-19T02:58:13.487 INFO:tasks.workunit.client.0.smithi092.stdout:./172/.snap/snap-subdir-test 2024-03-19T02:58:13.495 INFO:tasks.workunit.client.0.smithi092.stdout:./277/.snap/snap-subdir-test 2024-03-19T02:58:13.504 INFO:tasks.workunit.client.0.smithi092.stdout:./186/.snap/snap-subdir-test 2024-03-19T02:58:13.513 INFO:tasks.workunit.client.0.smithi092.stdout:./237/.snap/snap-subdir-test 2024-03-19T02:58:13.522 INFO:tasks.workunit.client.0.smithi092.stdout:./190/.snap/snap-subdir-test 2024-03-19T02:58:13.531 INFO:tasks.workunit.client.0.smithi092.stdout:./213/.snap/snap-subdir-test 2024-03-19T02:58:13.541 INFO:tasks.workunit.client.0.smithi092.stdout:./104/.snap/snap-subdir-test 2024-03-19T02:58:13.550 INFO:tasks.workunit.client.0.smithi092.stdout:./164/.snap/snap-subdir-test 2024-03-19T02:58:13.559 INFO:tasks.workunit.client.0.smithi092.stdout:./112/.snap/snap-subdir-test 2024-03-19T02:58:13.569 INFO:tasks.workunit.client.0.smithi092.stdout:./295/.snap/snap-subdir-test 2024-03-19T02:58:13.577 INFO:tasks.workunit.client.0.smithi092.stdout:./272/.snap/snap-subdir-test 2024-03-19T02:58:13.585 INFO:tasks.workunit.client.0.smithi092.stdout:./100/.snap/snap-subdir-test 2024-03-19T02:58:13.642 INFO:tasks.workunit.client.0.smithi092.stdout:./134/.snap/snap-subdir-test 2024-03-19T02:58:13.650 INFO:tasks.workunit.client.0.smithi092.stdout:./200/.snap/snap-subdir-test 2024-03-19T02:58:13.660 INFO:tasks.workunit.client.0.smithi092.stdout:./115/.snap/snap-subdir-test 2024-03-19T02:58:13.668 INFO:tasks.workunit.client.0.smithi092.stdout:./284/.snap/snap-subdir-test 2024-03-19T02:58:13.676 INFO:tasks.workunit.client.0.smithi092.stdout:./280/.snap/snap-subdir-test 2024-03-19T02:58:13.685 INFO:tasks.workunit.client.0.smithi092.stdout:./116/.snap/snap-subdir-test 2024-03-19T02:58:13.693 INFO:tasks.workunit.client.0.smithi092.stdout:./245/.snap/snap-subdir-test 2024-03-19T02:58:13.701 INFO:tasks.workunit.client.0.smithi092.stdout:./195/.snap/snap-subdir-test 2024-03-19T02:58:13.711 INFO:tasks.workunit.client.0.smithi092.stdout:./138/.snap/snap-subdir-test 2024-03-19T02:58:13.984 INFO:tasks.workunit.client.0.smithi092.stdout:./135/.snap/snap-subdir-test 2024-03-19T02:58:13.994 INFO:tasks.workunit.client.0.smithi092.stdout:./117/.snap/snap-subdir-test 2024-03-19T02:58:14.002 INFO:tasks.workunit.client.0.smithi092.stdout:./241/.snap/snap-subdir-test 2024-03-19T02:58:14.011 INFO:tasks.workunit.client.0.smithi092.stdout:./143/.snap/snap-subdir-test 2024-03-19T02:58:14.021 INFO:tasks.workunit.client.0.smithi092.stdout:./180/.snap/snap-subdir-test 2024-03-19T02:58:14.030 INFO:tasks.workunit.client.0.smithi092.stdout:./126/.snap/snap-subdir-test 2024-03-19T02:58:14.039 INFO:tasks.workunit.client.0.smithi092.stdout:./161/.snap/snap-subdir-test 2024-03-19T02:58:14.046 INFO:tasks.workunit.client.0.smithi092.stdout:./204/.snap/snap-subdir-test 2024-03-19T02:58:14.055 INFO:tasks.workunit.client.0.smithi092.stdout:./176/.snap/snap-subdir-test 2024-03-19T02:58:14.152 INFO:tasks.workunit.client.0.smithi092.stdout:./255/.snap/snap-subdir-test 2024-03-19T02:58:14.160 INFO:tasks.workunit.client.0.smithi092.stdout:./215/.snap/snap-subdir-test 2024-03-19T02:58:14.170 INFO:tasks.workunit.client.0.smithi092.stdout:./111/.snap/snap-subdir-test 2024-03-19T02:58:14.178 INFO:tasks.workunit.client.0.smithi092.stdout:./238/.snap/snap-subdir-test 2024-03-19T02:58:14.187 INFO:tasks.workunit.client.0.smithi092.stdout:./191/.snap/snap-subdir-test 2024-03-19T02:58:14.196 INFO:tasks.workunit.client.0.smithi092.stdout:./184/.snap/snap-subdir-test 2024-03-19T02:58:14.204 INFO:tasks.workunit.client.0.smithi092.stdout:./192/.snap/snap-subdir-test 2024-03-19T02:58:14.214 INFO:tasks.workunit.client.0.smithi092.stdout:./103/.snap/snap-subdir-test 2024-03-19T02:58:14.224 INFO:tasks.workunit.client.0.smithi092.stdout:./128/.snap/snap-subdir-test 2024-03-19T02:58:14.232 INFO:tasks.workunit.client.0.smithi092.stdout:./229/.snap/snap-subdir-test 2024-03-19T02:58:14.243 INFO:tasks.workunit.client.0.smithi092.stdout:./144/.snap/snap-subdir-test 2024-03-19T02:58:14.254 INFO:tasks.workunit.client.0.smithi092.stdout:./146/.snap/snap-subdir-test 2024-03-19T02:58:14.262 INFO:tasks.workunit.client.0.smithi092.stdout:./227/.snap/snap-subdir-test 2024-03-19T02:58:14.270 INFO:tasks.workunit.client.0.smithi092.stdout:./278/.snap/snap-subdir-test 2024-03-19T02:58:14.280 INFO:tasks.workunit.client.0.smithi092.stdout:./141/.snap/snap-subdir-test 2024-03-19T02:58:14.287 INFO:tasks.workunit.client.0.smithi092.stdout:./234/.snap/snap-subdir-test 2024-03-19T02:58:14.295 INFO:tasks.workunit.client.0.smithi092.stdout:./225/.snap/snap-subdir-test 2024-03-19T02:58:14.305 INFO:tasks.workunit.client.0.smithi092.stdout:./182/.snap/snap-subdir-test 2024-03-19T02:58:14.314 INFO:tasks.workunit.client.0.smithi092.stdout:./107/.snap/snap-subdir-test 2024-03-19T02:58:14.323 INFO:tasks.workunit.client.0.smithi092.stdout:./178/.snap/snap-subdir-test 2024-03-19T02:58:14.331 INFO:tasks.workunit.client.0.smithi092.stdout:./266/.snap/snap-subdir-test 2024-03-19T02:58:14.339 INFO:tasks.workunit.client.0.smithi092.stdout:./240/.snap/snap-subdir-test 2024-03-19T02:58:14.348 INFO:tasks.workunit.client.0.smithi092.stdout:./156/.snap/snap-subdir-test 2024-03-19T02:58:14.357 INFO:tasks.workunit.client.0.smithi092.stdout:./131/.snap/snap-subdir-test 2024-03-19T02:58:14.367 INFO:tasks.workunit.client.0.smithi092.stdout:./123/.snap/snap-subdir-test 2024-03-19T02:58:14.375 INFO:tasks.workunit.client.0.smithi092.stdout:./160/.snap/snap-subdir-test 2024-03-19T02:58:14.383 INFO:tasks.workunit.client.0.smithi092.stdout:./165/.snap/snap-subdir-test 2024-03-19T02:58:14.392 INFO:tasks.workunit.client.0.smithi092.stdout:./101/.snap/snap-subdir-test 2024-03-19T02:58:14.401 INFO:tasks.workunit.client.0.smithi092.stdout:./148/.snap/snap-subdir-test 2024-03-19T02:58:14.411 INFO:tasks.workunit.client.0.smithi092.stdout:./168/.snap/snap-subdir-test 2024-03-19T02:58:14.420 INFO:tasks.workunit.client.0.smithi092.stdout:./179/.snap/snap-subdir-test 2024-03-19T02:58:14.430 INFO:tasks.workunit.client.0.smithi092.stdout:./257/.snap/snap-subdir-test 2024-03-19T02:58:14.437 INFO:tasks.workunit.client.0.smithi092.stdout:./252/.snap/snap-subdir-test 2024-03-19T02:58:14.446 INFO:tasks.workunit.client.0.smithi092.stdout:./132/.snap/snap-subdir-test 2024-03-19T02:58:14.714 INFO:tasks.workunit.client.0.smithi092.stdout:./236/.snap/snap-subdir-test 2024-03-19T02:58:14.722 INFO:tasks.workunit.client.0.smithi092.stdout:./211/.snap/snap-subdir-test 2024-03-19T02:58:14.731 INFO:tasks.workunit.client.0.smithi092.stdout:./199/.snap/snap-subdir-test 2024-03-19T02:58:14.738 INFO:tasks.workunit.client.0.smithi092.stdout:./205/.snap/snap-subdir-test 2024-03-19T02:58:15.190 INFO:tasks.workunit.client.0.smithi092.stdout:./163/.snap/snap-subdir-test 2024-03-19T02:58:15.199 INFO:tasks.workunit.client.0.smithi092.stdout:./261/.snap/snap-subdir-test 2024-03-19T02:58:15.999 INFO:tasks.workunit.client.0.smithi092.stdout:./201/.snap/snap-subdir-test 2024-03-19T02:58:16.008 INFO:tasks.workunit.client.0.smithi092.stdout:./282/.snap/snap-subdir-test 2024-03-19T02:58:16.409 INFO:tasks.workunit.client.0.smithi092.stdout:./296/.snap/snap-subdir-test 2024-03-19T02:58:17.000 INFO:tasks.workunit.client.0.smithi092.stdout:./217/.snap/snap-subdir-test 2024-03-19T02:58:17.011 INFO:tasks.workunit.client.0.smithi092.stdout:./153/.snap/snap-subdir-test 2024-03-19T02:58:17.022 INFO:tasks.workunit.client.0.smithi092.stdout:./206/.snap/snap-subdir-test 2024-03-19T02:58:17.032 INFO:tasks.workunit.client.0.smithi092.stdout:./122/.snap/snap-subdir-test 2024-03-19T02:58:17.784 INFO:tasks.workunit.client.0.smithi092.stdout:./162/.snap/snap-subdir-test 2024-03-19T02:58:17.793 INFO:tasks.workunit.client.0.smithi092.stdout:./197/.snap/snap-subdir-test 2024-03-19T02:58:18.660 INFO:tasks.workunit.client.0.smithi092.stdout:./216/.snap/snap-subdir-test 2024-03-19T02:58:20.925 INFO:tasks.workunit.client.0.smithi092.stdout:./254/.snap/snap-subdir-test 2024-03-19T02:58:20.936 INFO:tasks.workunit.client.0.smithi092.stdout:./113/.snap/snap-subdir-test 2024-03-19T02:58:20.947 INFO:tasks.workunit.client.0.smithi092.stdout:./152/.snap/snap-subdir-test 2024-03-19T02:58:21.218 INFO:tasks.workunit.client.0.smithi092.stdout:./121/.snap/snap-subdir-test 2024-03-19T02:58:21.228 INFO:tasks.workunit.client.0.smithi092.stdout:./118/.snap/snap-subdir-test 2024-03-19T02:58:21.236 INFO:tasks.workunit.client.0.smithi092.stdout:./196/.snap/snap-subdir-test 2024-03-19T02:58:21.245 INFO:tasks.workunit.client.0.smithi092.stdout:./124/.snap/snap-subdir-test 2024-03-19T02:58:21.254 INFO:tasks.workunit.client.0.smithi092.stdout:./239/.snap/snap-subdir-test 2024-03-19T02:58:21.264 INFO:tasks.workunit.client.0.smithi092.stdout:./188/.snap/snap-subdir-test 2024-03-19T02:58:21.273 INFO:tasks.workunit.client.0.smithi092.stdout:./287/.snap/snap-subdir-test 2024-03-19T02:58:21.281 INFO:tasks.workunit.client.0.smithi092.stdout:./275/.snap/snap-subdir-test 2024-03-19T02:58:21.289 INFO:tasks.workunit.client.0.smithi092.stdout:./259/.snap/snap-subdir-test 2024-03-19T02:58:21.299 INFO:tasks.workunit.client.0.smithi092.stdout:./224/.snap/snap-subdir-test 2024-03-19T02:58:21.307 INFO:tasks.workunit.client.0.smithi092.stdout:./264/.snap/snap-subdir-test 2024-03-19T02:58:21.316 INFO:tasks.workunit.client.0.smithi092.stdout:./145/.snap/snap-subdir-test 2024-03-19T02:58:21.324 INFO:tasks.workunit.client.0.smithi092.stdout:./279/.snap/snap-subdir-test 2024-03-19T02:58:21.333 INFO:tasks.workunit.client.0.smithi092.stdout:./263/.snap/snap-subdir-test 2024-03-19T02:58:21.342 INFO:tasks.workunit.client.0.smithi092.stdout:./130/.snap/snap-subdir-test 2024-03-19T02:58:21.349 INFO:tasks.workunit.client.0.smithi092.stdout:./193/.snap/snap-subdir-test 2024-03-19T02:58:21.356 INFO:tasks.workunit.client.0.smithi092.stdout:./223/.snap/snap-subdir-test 2024-03-19T02:58:21.364 INFO:tasks.workunit.client.0.smithi092.stdout:./207/.snap/snap-subdir-test 2024-03-19T02:58:21.372 INFO:tasks.workunit.client.0.smithi092.stdout:./276/.snap/snap-subdir-test 2024-03-19T02:58:21.380 INFO:tasks.workunit.client.0.smithi092.stdout:./271/.snap/snap-subdir-test 2024-03-19T02:58:21.387 INFO:tasks.workunit.client.0.smithi092.stdout:./251/.snap/snap-subdir-test 2024-03-19T02:58:21.396 INFO:tasks.workunit.client.0.smithi092.stdout:./175/.snap/snap-subdir-test 2024-03-19T02:58:22.088 INFO:tasks.workunit.client.0.smithi092.stdout:./222/.snap/snap-subdir-test 2024-03-19T02:58:22.097 INFO:tasks.workunit.client.0.smithi092.stdout:./220/.snap/snap-subdir-test 2024-03-19T02:58:22.106 INFO:tasks.workunit.client.0.smithi092.stdout:./203/.snap/snap-subdir-test 2024-03-19T02:58:22.115 INFO:tasks.workunit.client.0.smithi092.stdout:./181/.snap/snap-subdir-test 2024-03-19T02:58:22.123 INFO:tasks.workunit.client.0.smithi092.stdout:./286/.snap/snap-subdir-test 2024-03-19T02:58:22.132 INFO:tasks.workunit.client.0.smithi092.stdout:./127/.snap/snap-subdir-test 2024-03-19T02:58:22.142 INFO:tasks.workunit.client.0.smithi092.stdout:./108/.snap/snap-subdir-test 2024-03-19T02:58:22.152 INFO:tasks.workunit.client.0.smithi092.stdout:./183/.snap/snap-subdir-test 2024-03-19T02:58:22.160 INFO:tasks.workunit.client.0.smithi092.stdout:./294/.snap/snap-subdir-test 2024-03-19T02:58:22.168 INFO:tasks.workunit.client.0.smithi092.stdout:./235/.snap/snap-subdir-test 2024-03-19T02:58:22.176 INFO:tasks.workunit.client.0.smithi092.stdout:./177/.snap/snap-subdir-test 2024-03-19T02:58:22.184 INFO:tasks.workunit.client.0.smithi092.stdout:./169/.snap/snap-subdir-test 2024-03-19T02:58:22.193 INFO:tasks.workunit.client.0.smithi092.stdout:./212/.snap/snap-subdir-test 2024-03-19T02:58:22.203 INFO:tasks.workunit.client.0.smithi092.stdout:./154/.snap/snap-subdir-test 2024-03-19T02:58:22.212 INFO:tasks.workunit.client.0.smithi092.stdout:./171/.snap/snap-subdir-test 2024-03-19T02:58:22.222 INFO:tasks.workunit.client.0.smithi092.stdout:./139/.snap/snap-subdir-test 2024-03-19T02:58:22.498 INFO:tasks.workunit.client.0.smithi092.stdout:./102/.snap/snap-subdir-test 2024-03-19T02:58:22.510 INFO:tasks.workunit.client.0.smithi092.stdout:./129/.snap/snap-subdir-test 2024-03-19T02:58:27.063 INFO:tasks.workunit.client.0.smithi092.stdout:Delete all the files and directories ... 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/19' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/15' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/31' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/36' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/20' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/12' 2024-03-19T02:58:28.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/39' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/10' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/34' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/38' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/11' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/16' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/32' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/30' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/14' 2024-03-19T02:58:28.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/18' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/13' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/33' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/17' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/35' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './100/37' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './100' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/19' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/15' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/31' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/36' 2024-03-19T02:58:28.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/20' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/12' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/39' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/10' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/34' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/38' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/11' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/16' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/32' 2024-03-19T02:58:28.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/30' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/14' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/18' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/13' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/33' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/17' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/35' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './101/37' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './101' 2024-03-19T02:58:28.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/19' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/15' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/31' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/36' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/20' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/12' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/39' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/10' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/34' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/38' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/11' 2024-03-19T02:58:28.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/16' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/32' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/30' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/14' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/18' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/13' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/33' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/17' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/35' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './102/37' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './102' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/19' 2024-03-19T02:58:28.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/15' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/31' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/36' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/20' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/12' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/39' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/10' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/34' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/38' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/11' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/16' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/32' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/30' 2024-03-19T02:58:28.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/14' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/18' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/13' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/33' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/17' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/35' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './103/37' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './103' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/19' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/15' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/31' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/36' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/20' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/12' 2024-03-19T02:58:28.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/39' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/10' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/34' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/38' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/11' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/16' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/32' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/30' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/14' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/18' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/13' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/33' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/17' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/35' 2024-03-19T02:58:28.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './104/37' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './104' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/19' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/15' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/31' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/36' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/20' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/12' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/39' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/10' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/34' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/38' 2024-03-19T02:58:28.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/11' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/16' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/32' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/30' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/14' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/18' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/13' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/33' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/17' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/35' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './105/37' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './105' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/19' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/15' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/31' 2024-03-19T02:58:28.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/36' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/20' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/12' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/39' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/10' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/34' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/38' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/11' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/16' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/32' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/30' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/14' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/18' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/13' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/33' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/17' 2024-03-19T02:58:28.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/35' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './106/37' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './106' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/19' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/15' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/31' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/36' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/20' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/12' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/39' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/10' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/34' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/38' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/11' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/16' 2024-03-19T02:58:28.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/32' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/30' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/14' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/18' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/13' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/33' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/17' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/35' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './107/37' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './107' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/19' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/15' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/31' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/36' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/20' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/12' 2024-03-19T02:58:28.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/39' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/10' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/34' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/38' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/11' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/16' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/32' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/30' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/14' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/18' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/13' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/33' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/17' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/35' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './108/37' 2024-03-19T02:58:28.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './108' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/19' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/15' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/31' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/36' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/20' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/12' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/39' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/10' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/34' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/38' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/11' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/16' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/32' 2024-03-19T02:58:28.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/30' 2024-03-19T02:58:29.698 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/14' 2024-03-19T02:58:29.698 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/18' 2024-03-19T02:58:29.698 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/13' 2024-03-19T02:58:29.698 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/33' 2024-03-19T02:58:29.698 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/17' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/35' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './109/37' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './109' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/19' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/15' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/31' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/36' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/20' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/12' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/39' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/10' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/34' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/38' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/11' 2024-03-19T02:58:29.699 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/16' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/32' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/30' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/14' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/18' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/13' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/33' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/17' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/35' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './110/37' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './110' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/19' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/15' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/31' 2024-03-19T02:58:29.700 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/36' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/20' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/12' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/39' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/10' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/34' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/38' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/11' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/16' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/32' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/30' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/14' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/18' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/13' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/33' 2024-03-19T02:58:29.701 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/17' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/35' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './111/37' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './111' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/19' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/15' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/31' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/36' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/20' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/12' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/39' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/10' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/34' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/38' 2024-03-19T02:58:29.702 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/11' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/16' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/32' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/30' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/14' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/18' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/13' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/33' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/17' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/35' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './112/37' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './112' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/19' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/15' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/31' 2024-03-19T02:58:29.703 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/36' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/20' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/12' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/39' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/10' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/34' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/38' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/11' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/16' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/32' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/30' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/14' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/18' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/13' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/33' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/17' 2024-03-19T02:58:29.704 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/35' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './113/37' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './113' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/19' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/15' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/31' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/36' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/20' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/12' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/39' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/10' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/34' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/38' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/11' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/16' 2024-03-19T02:58:29.705 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/32' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/30' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/14' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/18' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/13' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/33' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/17' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/35' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './114/37' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './114' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/19' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/15' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/31' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/36' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/20' 2024-03-19T02:58:29.706 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/12' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/39' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/10' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/34' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/38' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/11' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/16' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/32' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/30' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/14' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/18' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/13' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/33' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/17' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/35' 2024-03-19T02:58:29.707 INFO:tasks.workunit.client.0.smithi092.stdout:removed './115/37' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './115' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/19' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/15' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/31' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/36' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/20' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/12' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/39' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/10' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/34' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/38' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/11' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/16' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/32' 2024-03-19T02:58:29.708 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/30' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/14' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/18' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/13' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/33' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/17' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/35' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './116/37' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './116' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/19' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/15' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/31' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/36' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/20' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/12' 2024-03-19T02:58:29.709 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/39' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/10' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/34' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/38' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/11' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/16' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/32' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/30' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/14' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/18' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/13' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/33' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/17' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/35' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed './117/37' 2024-03-19T02:58:29.710 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './117' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/19' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/15' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/31' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/36' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/20' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/12' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/39' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/10' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/34' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/38' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/11' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/16' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/32' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/30' 2024-03-19T02:58:29.711 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/14' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/18' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/13' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/33' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/17' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/35' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './118/37' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './118' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/19' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/15' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/31' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/36' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/20' 2024-03-19T02:58:29.712 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/12' 2024-03-19T02:58:29.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:58:29.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/39' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/10' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/34' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/38' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/11' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/16' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/32' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/30' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/14' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/18' 2024-03-19T02:58:30.842 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/13' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/33' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/17' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/35' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './119/37' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './119' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/19' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/15' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/31' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/36' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/20' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/12' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/39' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/10' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/34' 2024-03-19T02:58:30.843 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/38' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/11' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/16' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/32' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/30' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/14' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/18' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/13' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/33' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/17' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/35' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './120/37' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './120' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/19' 2024-03-19T02:58:30.844 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/15' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/31' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/36' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/20' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/12' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/39' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/10' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/34' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/38' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/11' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/16' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/32' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/30' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/14' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/18' 2024-03-19T02:58:30.845 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/13' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/33' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/17' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/35' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './121/37' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './121' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/19' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/15' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/31' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/36' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/20' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/12' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/39' 2024-03-19T02:58:30.846 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/10' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/34' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/38' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/11' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/16' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/32' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/30' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/14' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/18' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/13' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/33' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/17' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/35' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './122/37' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './122' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/19' 2024-03-19T02:58:30.847 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/15' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/31' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/36' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/20' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/12' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/39' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/10' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/34' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/38' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/11' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/16' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/32' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/30' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/14' 2024-03-19T02:58:30.848 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/18' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/13' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/33' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/17' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/35' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './123/37' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './123' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/19' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/15' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/31' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/36' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/20' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/12' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/39' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/10' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/34' 2024-03-19T02:58:30.849 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/38' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/11' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/16' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/32' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/30' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/14' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/18' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/13' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/33' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/17' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/35' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './124/37' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './124' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/19' 2024-03-19T02:58:30.850 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/15' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/31' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/36' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/20' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/12' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/39' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/10' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/34' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/38' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/11' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/16' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/32' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/30' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/14' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/18' 2024-03-19T02:58:30.851 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/13' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/33' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/17' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/35' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './125/37' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './125' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/19' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/15' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/31' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/36' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/20' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/12' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/39' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/10' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/34' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/38' 2024-03-19T02:58:30.852 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/11' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/16' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/32' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/30' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/14' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/18' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/13' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/33' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/17' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/35' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './126/37' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './126' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/19' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/15' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/31' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/36' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/20' 2024-03-19T02:58:30.853 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/12' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/39' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/10' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/34' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/38' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/11' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/16' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/32' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/30' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/14' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/18' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/13' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/33' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/17' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/35' 2024-03-19T02:58:30.854 INFO:tasks.workunit.client.0.smithi092.stdout:removed './127/37' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './127' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/19' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/15' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/31' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/36' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/20' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/12' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/39' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/10' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/34' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/38' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/11' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/16' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/32' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/30' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/14' 2024-03-19T02:58:30.855 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/18' 2024-03-19T02:58:30.856 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/13' 2024-03-19T02:58:30.856 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/33' 2024-03-19T02:58:30.856 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/17' 2024-03-19T02:58:30.856 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/35' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './128/37' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './128' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/19' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/15' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/31' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/36' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/20' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/12' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/39' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/10' 2024-03-19T02:58:32.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/34' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/38' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/11' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/16' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/32' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/30' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/14' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/18' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/13' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/33' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/17' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/35' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './129/37' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './129' 2024-03-19T02:58:32.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/19' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/15' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/31' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/36' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/20' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/12' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/39' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/10' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/34' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/38' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/11' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/16' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/32' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/30' 2024-03-19T02:58:32.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/14' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/18' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/13' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/33' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/17' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/35' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './130/37' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './130' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/19' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/15' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/31' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/36' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/20' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/12' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/39' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/10' 2024-03-19T02:58:32.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/34' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/38' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/11' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/16' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/32' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/30' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/14' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/18' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/13' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/33' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/17' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/35' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './131/37' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './131' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/19' 2024-03-19T02:58:32.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/15' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/31' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/36' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/20' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/12' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/39' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/10' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/34' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/38' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/11' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/16' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/32' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/30' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/14' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/18' 2024-03-19T02:58:32.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/13' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/33' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/17' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/35' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './132/37' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './132' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/19' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/15' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/31' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/36' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/20' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/12' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/39' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/10' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/34' 2024-03-19T02:58:32.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/38' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/11' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/16' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/32' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/30' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/14' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/18' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/13' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/33' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/17' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/35' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './133/37' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './133' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/19' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/15' 2024-03-19T02:58:32.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/31' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/36' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/20' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/12' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/39' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/10' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/34' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/38' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/11' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/16' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/32' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/30' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/14' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/18' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/13' 2024-03-19T02:58:32.740 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/33' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/17' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/35' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './134/37' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './134' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/19' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/15' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/31' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/36' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/20' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/12' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/39' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/10' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/34' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/38' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/11' 2024-03-19T02:58:32.741 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/16' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/32' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/30' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/14' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/18' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/13' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/33' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/17' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/35' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './135/37' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './135' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/19' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/15' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/31' 2024-03-19T02:58:32.742 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/36' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/20' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/12' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/39' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/10' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/34' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/38' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/11' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/16' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/32' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/30' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/14' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/18' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/13' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/33' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/17' 2024-03-19T02:58:32.743 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/35' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './136/37' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './136' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/19' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/15' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/31' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/36' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/20' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/12' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/39' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/10' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/34' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/38' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/11' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/16' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/32' 2024-03-19T02:58:32.744 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/30' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/14' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/18' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/13' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/33' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/17' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/35' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './137/37' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './137' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/19' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/15' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/31' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/36' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/20' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/12' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/39' 2024-03-19T02:58:32.745 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/10' 2024-03-19T02:58:32.746 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/34' 2024-03-19T02:58:32.746 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/38' 2024-03-19T02:58:32.746 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/11' 2024-03-19T02:58:32.746 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/16' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/32' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/30' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/14' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/18' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/13' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/33' 2024-03-19T02:58:33.411 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/17' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/35' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './138/37' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './138' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/19' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/15' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/31' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/36' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/20' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/12' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/39' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/10' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/34' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/38' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/11' 2024-03-19T02:58:33.412 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/16' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/32' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/30' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/14' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/18' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/13' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/33' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/17' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/35' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './139/37' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './139' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/19' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/15' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/31' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/36' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/20' 2024-03-19T02:58:33.413 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/12' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/39' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/10' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/34' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/38' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/11' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/16' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/32' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/30' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/14' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/18' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/13' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/33' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/17' 2024-03-19T02:58:33.414 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/35' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './140/37' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './140' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/19' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/15' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/31' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/36' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/20' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/12' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/39' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/10' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/34' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/38' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/11' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/16' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/32' 2024-03-19T02:58:33.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/30' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/14' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/18' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/13' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/33' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/17' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/35' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './141/37' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './141' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/19' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/15' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/31' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/36' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/20' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/12' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/39' 2024-03-19T02:58:33.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/10' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/34' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/38' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/11' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/16' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/32' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/30' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/14' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/18' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/13' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/33' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/17' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/35' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './142/37' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './142' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/19' 2024-03-19T02:58:33.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/15' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/31' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/36' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/20' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/12' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/39' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/10' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/34' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/38' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/11' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/16' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/32' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/30' 2024-03-19T02:58:33.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/14' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/18' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/13' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/33' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/17' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/35' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './143/37' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './143' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/19' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/15' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/31' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/36' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/20' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/12' 2024-03-19T02:58:33.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/39' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/10' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/34' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/38' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/11' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/16' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/32' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/30' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/14' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/18' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/13' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/33' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/17' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/35' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './144/37' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './144' 2024-03-19T02:58:33.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/19' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/15' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/31' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/36' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/20' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/12' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/39' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/10' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/34' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/38' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/11' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/16' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/32' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/30' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/14' 2024-03-19T02:58:33.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/18' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/13' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/33' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/17' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/35' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './145/37' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './145' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/19' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/15' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/31' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/36' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/20' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/12' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/39' 2024-03-19T02:58:33.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/10' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/34' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/38' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/11' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/16' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/32' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/30' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/14' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/18' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/13' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/33' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/17' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/35' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './146/37' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './146' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/19' 2024-03-19T02:58:33.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/15' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/31' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/36' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/20' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/12' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/39' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/10' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/34' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/38' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/11' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/16' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/32' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/30' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/14' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/18' 2024-03-19T02:58:33.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/13' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/33' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/17' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/35' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './147/37' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './147' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/19' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/15' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/31' 2024-03-19T02:58:33.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/36' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/20' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/12' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/39' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/10' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/34' 2024-03-19T02:58:34.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/38' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/11' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/16' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/32' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/30' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/14' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/18' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/13' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/33' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/17' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/35' 2024-03-19T02:58:34.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './148/37' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './148' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/19' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/15' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/31' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/36' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/20' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/12' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/39' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/10' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/34' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/38' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/11' 2024-03-19T02:58:34.067 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/16' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/32' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/30' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/14' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/18' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/13' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/33' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/17' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/35' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './149/37' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './149' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/19' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/15' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/31' 2024-03-19T02:58:34.068 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/36' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/20' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/12' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/39' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/10' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/34' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/38' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/11' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/16' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/32' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/30' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/14' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/18' 2024-03-19T02:58:34.069 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/13' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/33' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/17' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/35' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './150/37' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './150' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/19' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/15' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/31' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/36' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/20' 2024-03-19T02:58:34.070 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/12' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/39' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/10' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/34' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/38' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/11' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/16' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/32' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/30' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/14' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/18' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/13' 2024-03-19T02:58:34.071 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/33' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/17' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/35' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './151/37' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './151' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/19' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/15' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/31' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/36' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/20' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/12' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/39' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/10' 2024-03-19T02:58:34.072 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/34' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/38' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/11' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/16' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/32' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/30' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/14' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/18' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/13' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/33' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/17' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/35' 2024-03-19T02:58:34.073 INFO:tasks.workunit.client.0.smithi092.stdout:removed './152/37' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './152' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/19' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/15' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/31' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/36' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/20' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/12' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/39' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/10' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/34' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/38' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/11' 2024-03-19T02:58:34.074 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/16' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/32' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/30' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/14' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/18' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/13' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/33' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/17' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/35' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './153/37' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './153' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/19' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/15' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/31' 2024-03-19T02:58:34.075 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/36' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/20' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/12' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/39' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/10' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/34' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/38' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/11' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/16' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/32' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/30' 2024-03-19T02:58:34.076 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/14' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/18' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/13' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/33' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/17' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/35' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './154/37' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './154' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/19' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/15' 2024-03-19T02:58:34.077 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/31' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/36' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/20' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/12' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/39' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/10' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/34' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/38' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/11' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/16' 2024-03-19T02:58:34.078 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/32' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/30' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/14' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/18' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/13' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/33' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/17' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/35' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './155/37' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './155' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/19' 2024-03-19T02:58:34.079 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/15' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/31' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/36' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/20' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/12' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/39' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/10' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/34' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/38' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/11' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/16' 2024-03-19T02:58:34.080 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/32' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/30' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/14' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/18' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/13' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/33' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/17' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/35' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './156/37' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './156' 2024-03-19T02:58:34.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/19' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/15' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/31' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/36' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/20' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/12' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/39' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/10' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/34' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/38' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/11' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/16' 2024-03-19T02:58:34.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/32' 2024-03-19T02:58:34.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/30' 2024-03-19T02:58:34.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/14' 2024-03-19T02:58:34.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/18' 2024-03-19T02:58:34.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/13' 2024-03-19T02:58:34.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/33' 2024-03-19T02:58:35.491 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/17' 2024-03-19T02:58:35.491 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/35' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './157/37' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './157' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/19' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/15' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/31' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/36' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/20' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/12' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/39' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/10' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/34' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/38' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/11' 2024-03-19T02:58:35.492 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/16' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/32' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/30' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/14' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/18' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/13' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/33' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/17' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/35' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './158/37' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './158' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/19' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/15' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/31' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/36' 2024-03-19T02:58:35.493 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/20' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/12' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/39' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/10' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/34' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/38' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/11' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/16' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/32' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/30' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/14' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/18' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/13' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/33' 2024-03-19T02:58:35.494 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/17' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/35' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './159/37' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './159' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/19' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/15' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/31' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/36' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/20' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/12' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/39' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/10' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/34' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/38' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/11' 2024-03-19T02:58:35.495 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/16' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/32' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/30' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/14' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/18' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/13' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/33' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/17' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/35' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './160/37' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './160' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/19' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/15' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/31' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/36' 2024-03-19T02:58:35.496 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/20' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/12' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/39' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/10' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/34' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/38' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/11' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/16' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/32' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/30' 2024-03-19T02:58:35.497 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/14' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/18' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/13' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/33' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/17' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/35' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './161/37' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './161' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/19' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/15' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/31' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/36' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/20' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/12' 2024-03-19T02:58:35.498 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/39' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/10' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/34' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/38' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/11' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/16' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/32' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/30' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/14' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/18' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/13' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/33' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/17' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/35' 2024-03-19T02:58:35.499 INFO:tasks.workunit.client.0.smithi092.stdout:removed './162/37' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './162' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/19' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/15' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/31' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/36' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/20' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/12' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/39' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/10' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/34' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/38' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/11' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/16' 2024-03-19T02:58:35.500 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/32' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/30' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/14' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/18' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/13' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/33' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/17' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/35' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './163/37' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './163' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/19' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/15' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/31' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/36' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/20' 2024-03-19T02:58:35.501 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/12' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/39' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/10' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/34' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/38' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/11' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/16' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/32' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/30' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/14' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/18' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/13' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/33' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/17' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/35' 2024-03-19T02:58:35.502 INFO:tasks.workunit.client.0.smithi092.stdout:removed './164/37' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './164' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/19' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/15' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/31' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/36' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/20' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/12' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/39' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/10' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/34' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/38' 2024-03-19T02:58:35.503 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/11' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/16' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/32' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/30' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/14' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/18' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/13' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/33' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/17' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/35' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './165/37' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './165' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/19' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/15' 2024-03-19T02:58:35.504 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/31' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/36' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/20' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/12' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/39' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/10' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/34' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/38' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/11' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/16' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/32' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/30' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/14' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/18' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/13' 2024-03-19T02:58:35.505 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/33' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/17' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/35' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './166/37' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './166' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/19' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/15' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/31' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/36' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/20' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/12' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/39' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/10' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/34' 2024-03-19T02:58:35.506 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/38' 2024-03-19T02:58:37.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/11' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/16' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/32' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/30' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/14' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/18' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/13' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/33' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/17' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/35' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './167/37' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './167' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/19' 2024-03-19T02:58:37.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/15' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/31' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/36' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/20' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/12' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/39' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/10' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/34' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/38' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/11' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/16' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/32' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/30' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/14' 2024-03-19T02:58:37.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/18' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/13' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/33' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/17' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/35' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './168/37' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './168' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/19' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/15' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/31' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/36' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/20' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/12' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/39' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/10' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/34' 2024-03-19T02:58:37.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/38' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/11' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/16' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/32' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/30' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/14' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/18' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/13' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/33' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/17' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/35' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './169/37' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './169' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/19' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/15' 2024-03-19T02:58:37.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/31' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/36' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/20' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/12' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/39' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/10' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/34' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/38' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/11' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/16' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/32' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/30' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/14' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/18' 2024-03-19T02:58:37.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/13' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/33' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/17' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/35' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './170/37' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './170' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/19' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/15' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/31' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/36' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/20' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/12' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/39' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/10' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/34' 2024-03-19T02:58:37.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/38' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/11' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/16' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/32' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/30' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/14' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/18' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/13' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/33' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/17' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/35' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './171/37' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './171' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/19' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/15' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/31' 2024-03-19T02:58:37.156 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/36' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/20' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/12' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/39' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/10' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/34' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/38' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/11' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/16' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/32' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/30' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/14' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/18' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/13' 2024-03-19T02:58:37.157 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/33' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/17' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/35' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './172/37' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './172' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/19' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/15' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/31' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/36' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/20' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/12' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/39' 2024-03-19T02:58:37.158 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/10' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/34' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/38' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/11' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/16' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/32' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/30' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/14' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/18' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/13' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/33' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/17' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/35' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './173/37' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './173' 2024-03-19T02:58:37.159 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/19' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/15' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/31' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/36' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/20' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/12' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/39' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/10' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/34' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/38' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/11' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/16' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/32' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/30' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/14' 2024-03-19T02:58:37.160 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/18' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/13' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/33' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/17' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/35' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './174/37' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './174' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/19' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/15' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/31' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/36' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/20' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/12' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/39' 2024-03-19T02:58:37.161 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/10' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/34' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/38' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/11' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/16' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/32' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/30' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/14' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/18' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/13' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/33' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/17' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/35' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed './175/37' 2024-03-19T02:58:37.162 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './175' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/19' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/15' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/31' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/36' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/20' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/12' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/39' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/10' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/34' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/38' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/11' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/16' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/32' 2024-03-19T02:58:37.163 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/30' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/14' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/18' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/13' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/33' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/17' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/35' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './176/37' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './176' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/19' 2024-03-19T02:58:37.164 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/15' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/31' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/36' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/20' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/12' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/39' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/10' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/34' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/38' 2024-03-19T02:58:39.946 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/11' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/16' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/32' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/30' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/14' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/18' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/13' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/33' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/17' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/35' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './177/37' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './177' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/19' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/15' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/31' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/36' 2024-03-19T02:58:39.947 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/20' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/12' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/39' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/10' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/34' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/38' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/11' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/16' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/32' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/30' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/14' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/18' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/13' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/33' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/17' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/35' 2024-03-19T02:58:39.948 INFO:tasks.workunit.client.0.smithi092.stdout:removed './178/37' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './178' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/19' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/15' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/31' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/36' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/20' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/12' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/39' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/10' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/34' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/38' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/11' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/16' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/32' 2024-03-19T02:58:39.949 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/30' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/14' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/18' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/13' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/33' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/17' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/35' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './179/37' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './179' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/19' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/15' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/31' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/36' 2024-03-19T02:58:39.950 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/20' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/12' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/39' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/10' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/34' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/38' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/11' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/16' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/32' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/30' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/14' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/18' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/13' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/33' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/17' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/35' 2024-03-19T02:58:39.951 INFO:tasks.workunit.client.0.smithi092.stdout:removed './180/37' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './180' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/19' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/15' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/31' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/36' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/20' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/12' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/39' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/10' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/34' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/38' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/11' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/16' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/32' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/30' 2024-03-19T02:58:39.952 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/14' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/18' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/13' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/33' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/17' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/35' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './181/37' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './181' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/19' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/15' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/31' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/36' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/20' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/12' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/39' 2024-03-19T02:58:39.953 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/10' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/34' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/38' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/11' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/16' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/32' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/30' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/14' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/18' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/13' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/33' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/17' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/35' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed './182/37' 2024-03-19T02:58:39.954 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './182' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/19' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/15' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/31' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/36' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/20' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/12' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/39' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/10' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/34' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/38' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/11' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/16' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/32' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/30' 2024-03-19T02:58:39.955 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/14' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/18' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/13' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/33' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/17' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/35' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './183/37' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './183' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/19' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/15' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/31' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/36' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/20' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/12' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/39' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/10' 2024-03-19T02:58:39.956 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/34' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/38' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/11' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/16' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/32' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/30' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/14' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/18' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/13' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/33' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/17' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/35' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './184/37' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './184' 2024-03-19T02:58:39.957 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/19' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/15' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/31' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/36' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/20' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/12' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/39' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/10' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/34' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/38' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/11' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/16' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/32' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/30' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/14' 2024-03-19T02:58:39.958 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/18' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/13' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/33' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/17' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/35' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './185/37' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './185' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/19' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/15' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/31' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/36' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/20' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/12' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/39' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/10' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/34' 2024-03-19T02:58:39.959 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/38' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/11' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/16' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/32' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/30' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/14' 2024-03-19T02:58:39.960 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/18' 2024-03-19T02:58:40.453 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/13' 2024-03-19T02:58:40.453 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/33' 2024-03-19T02:58:40.453 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/17' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/35' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './186/37' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './186' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/19' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/15' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/31' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/36' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/20' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/12' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/39' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/10' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/34' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/38' 2024-03-19T02:58:40.454 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/11' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/16' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/32' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/30' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/14' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/18' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/13' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/33' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/17' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/35' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './187/37' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './187' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/19' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/15' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/31' 2024-03-19T02:58:40.455 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/36' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/20' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/12' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/39' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/10' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/34' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/38' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/11' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/16' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/32' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/30' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/14' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/18' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/13' 2024-03-19T02:58:40.456 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/33' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/17' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/35' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './188/37' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './188' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/19' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/15' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/31' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/36' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/20' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/12' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/39' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/10' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/34' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/38' 2024-03-19T02:58:40.457 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/11' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/16' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/32' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/30' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/14' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/18' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/13' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/33' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/17' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/35' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './189/37' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './189' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/19' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/15' 2024-03-19T02:58:40.458 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/31' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/36' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/20' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/12' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/39' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/10' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/34' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/38' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/11' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/16' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/32' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/30' 2024-03-19T02:58:40.459 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/14' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/18' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/13' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/33' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/17' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/35' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './190/37' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './190' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/19' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/15' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/31' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/36' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/20' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/12' 2024-03-19T02:58:40.460 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/39' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/10' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/34' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/38' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/11' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/16' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/32' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/30' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/14' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/18' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/13' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/33' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/17' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/35' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './191/37' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './191' 2024-03-19T02:58:40.461 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/19' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/15' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/31' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/36' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/20' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/12' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/39' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/10' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/34' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/38' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/11' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/16' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/32' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/30' 2024-03-19T02:58:40.462 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/14' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/18' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/13' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/33' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/17' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/35' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './192/37' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './192' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/19' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/15' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/31' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/36' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/20' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/12' 2024-03-19T02:58:40.463 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/39' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/10' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/34' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/38' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/11' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/16' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/32' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/30' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/14' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/18' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/13' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/33' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/17' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/35' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed './193/37' 2024-03-19T02:58:40.464 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './193' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/19' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/15' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/31' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/36' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/20' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/12' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/39' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/10' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/34' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/38' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/11' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/16' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/32' 2024-03-19T02:58:40.465 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/30' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/14' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/18' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/13' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/33' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/17' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/35' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './194/37' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './194' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/19' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/15' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/31' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/36' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/20' 2024-03-19T02:58:40.466 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/12' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/39' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/10' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/34' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/38' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/11' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/16' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/32' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/30' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/14' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/18' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/13' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/33' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/17' 2024-03-19T02:58:40.467 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/35' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './195/37' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './195' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/19' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/15' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/31' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/36' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/20' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/12' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/39' 2024-03-19T02:58:40.468 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/10' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/34' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/38' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/11' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/16' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/32' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/30' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/14' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/18' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/13' 2024-03-19T02:58:42.027 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/33' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/17' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/35' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './196/37' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './196' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/19' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/15' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/31' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/36' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/20' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/12' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/39' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/10' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/34' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/38' 2024-03-19T02:58:42.028 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/11' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/16' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/32' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/30' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/14' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/18' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/13' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/33' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/17' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/35' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './197/37' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './197' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/19' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/15' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/31' 2024-03-19T02:58:42.029 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/36' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/20' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/12' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/39' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/10' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/34' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/38' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/11' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/16' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/32' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/30' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/14' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/18' 2024-03-19T02:58:42.030 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/13' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/33' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/17' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/35' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './198/37' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './198' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/19' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/15' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/31' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/36' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/20' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/12' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/39' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/10' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/34' 2024-03-19T02:58:42.031 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/38' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/11' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/16' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/32' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/30' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/14' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/18' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/13' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/33' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/17' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/35' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './199/37' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './199' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/21' 2024-03-19T02:58:42.032 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/27' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/31' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/36' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/20' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/39' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/34' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/38' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/28' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/24' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/29' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/32' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/22' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/25' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/30' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/33' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/23' 2024-03-19T02:58:42.033 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/35' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/26' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './200/37' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './200' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/21' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/27' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/31' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/36' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/20' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/39' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/34' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/38' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/28' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/24' 2024-03-19T02:58:42.034 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/29' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/32' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/22' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/25' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/30' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/33' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/23' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/35' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/26' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './201/37' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './201' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/21' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/27' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/31' 2024-03-19T02:58:42.035 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/36' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/20' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/39' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/34' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/38' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/28' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/24' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/29' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/32' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/22' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/25' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/30' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/33' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/23' 2024-03-19T02:58:42.036 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/35' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/26' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './202/37' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './202' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/21' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/27' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/31' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/36' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/20' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/39' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/34' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/38' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/28' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/24' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/29' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/32' 2024-03-19T02:58:42.037 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/22' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/25' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/30' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/33' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/23' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/35' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/26' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './203/37' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './203' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/21' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/27' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/31' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/36' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/20' 2024-03-19T02:58:42.038 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/39' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/34' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/38' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/28' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/24' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/29' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/32' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/22' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/25' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/30' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/33' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/23' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/35' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/26' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed './204/37' 2024-03-19T02:58:42.039 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './204' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/21' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/27' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/31' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/36' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/20' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/39' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/34' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/38' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/28' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/24' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/29' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/32' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/22' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/25' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/30' 2024-03-19T02:58:42.040 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/33' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/23' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/35' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/26' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './205/37' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './205' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/21' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/27' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/31' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/36' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/20' 2024-03-19T02:58:42.041 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/39' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/34' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/38' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/28' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/24' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/29' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/32' 2024-03-19T02:58:44.309 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/22' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/25' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/30' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/33' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/23' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/35' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/26' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './206/37' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './206' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/21' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/27' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/31' 2024-03-19T02:58:44.310 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/36' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/20' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/39' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/34' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/38' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/28' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/24' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/29' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/32' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/22' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/25' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/30' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/33' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/23' 2024-03-19T02:58:44.311 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/35' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/26' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './207/37' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './207' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/21' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/27' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/31' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/36' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/20' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/39' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/34' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/38' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/28' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/24' 2024-03-19T02:58:44.312 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/29' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/32' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/22' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/25' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/30' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/33' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/23' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/35' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/26' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './208/37' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './208' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/21' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/27' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/31' 2024-03-19T02:58:44.313 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/36' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/20' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/39' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/34' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/38' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/28' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/24' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/29' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/32' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/22' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/25' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/30' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/33' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/23' 2024-03-19T02:58:44.314 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/35' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/26' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './209/37' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './209' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/21' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/27' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/31' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/36' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/20' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/39' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/34' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/38' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/28' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/24' 2024-03-19T02:58:44.315 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/29' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/32' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/22' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/25' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/30' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/33' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/23' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/35' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/26' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './210/37' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './210' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/21' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/27' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/31' 2024-03-19T02:58:44.316 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/36' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/20' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/39' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/34' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/38' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/28' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/24' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/29' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/32' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/22' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/25' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/30' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/33' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/23' 2024-03-19T02:58:44.317 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/35' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/26' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './211/37' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './211' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/21' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/27' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/31' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/36' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/20' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/39' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/34' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/38' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/28' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/24' 2024-03-19T02:58:44.318 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/29' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/32' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/22' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/25' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/30' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/33' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/23' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/35' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/26' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './212/37' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './212' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/21' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/27' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/31' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/36' 2024-03-19T02:58:44.319 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/20' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/39' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/34' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/38' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/28' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/24' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/29' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/32' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/22' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/25' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/30' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/33' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/23' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/35' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/26' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed './213/37' 2024-03-19T02:58:44.320 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './213' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/21' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/27' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/31' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/36' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/20' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/39' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/34' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/38' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/28' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/24' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/29' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/32' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/22' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/25' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/30' 2024-03-19T02:58:44.321 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/33' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/23' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/35' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/26' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './214/37' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './214' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/21' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/27' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/31' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/36' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/20' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/39' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/34' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/38' 2024-03-19T02:58:44.322 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/28' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/24' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/29' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/32' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/22' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/25' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/30' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/33' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/23' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/35' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/26' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './215/37' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './215' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/21' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/27' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/31' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/36' 2024-03-19T02:58:44.323 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/20' 2024-03-19T02:58:44.324 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/39' 2024-03-19T02:58:44.324 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/34' 2024-03-19T02:58:44.324 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/38' 2024-03-19T02:58:45.965 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/28' 2024-03-19T02:58:45.965 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/24' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/29' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/32' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/22' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/25' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/30' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/33' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/23' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/35' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/26' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './216/37' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './216' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/21' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/27' 2024-03-19T02:58:45.966 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/31' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/36' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/20' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/39' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/34' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/38' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/28' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/24' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/29' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/32' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/22' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/25' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/30' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/33' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/23' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/35' 2024-03-19T02:58:45.967 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/26' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './217/37' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './217' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/21' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/27' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/31' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/36' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/20' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/39' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/34' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/38' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/28' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/24' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/29' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/32' 2024-03-19T02:58:45.968 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/22' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/25' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/30' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/33' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/23' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/35' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/26' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './218/37' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './218' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/21' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/27' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/31' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/36' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/20' 2024-03-19T02:58:45.969 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/39' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/34' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/38' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/28' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/24' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/29' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/32' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/22' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/25' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/30' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/33' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/23' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/35' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/26' 2024-03-19T02:58:45.970 INFO:tasks.workunit.client.0.smithi092.stdout:removed './219/37' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './219' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/21' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/27' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/31' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/36' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/20' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/39' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/34' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/38' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/28' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/24' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/29' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/32' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/22' 2024-03-19T02:58:45.971 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/25' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/30' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/33' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/23' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/35' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/26' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './220/37' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './220' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/21' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/27' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/31' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/36' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/20' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/39' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/34' 2024-03-19T02:58:45.972 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/38' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/28' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/24' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/29' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/32' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/22' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/25' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/30' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/33' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/23' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/35' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/26' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './221/37' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './221' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/21' 2024-03-19T02:58:45.973 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/27' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/31' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/36' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/20' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/39' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/34' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/38' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/28' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/24' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/29' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/32' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/22' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/25' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/30' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/33' 2024-03-19T02:58:45.974 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/23' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/35' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/26' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './222/37' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './222' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/21' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/27' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/31' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/36' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/20' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/39' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/34' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/38' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/28' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/24' 2024-03-19T02:58:45.975 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/29' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/32' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/22' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/25' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/30' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/33' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/23' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/35' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/26' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './223/37' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './223' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/21' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/27' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/31' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/36' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/20' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/39' 2024-03-19T02:58:45.976 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/34' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/38' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/28' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/24' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/29' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/32' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/22' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/25' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/30' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/33' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/23' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/35' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/26' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './224/37' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './224' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/21' 2024-03-19T02:58:45.977 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/27' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/31' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/36' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/20' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/39' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/34' 2024-03-19T02:58:45.978 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/38' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/28' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/24' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/29' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/32' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/22' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/25' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/30' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/33' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/23' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/35' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/26' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './225/37' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './225' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/21' 2024-03-19T02:58:45.979 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/27' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/31' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/36' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/20' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/39' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/34' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/38' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/28' 2024-03-19T02:58:45.980 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/24' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/29' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/32' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/22' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/25' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/30' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/33' 2024-03-19T02:58:48.565 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/23' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/35' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/26' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './226/37' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './226' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/21' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/27' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/31' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/36' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/20' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/39' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/34' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/38' 2024-03-19T02:58:48.566 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/28' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/24' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/29' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/32' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/22' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/25' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/30' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/33' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/23' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/35' 2024-03-19T02:58:48.567 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/26' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './227/37' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './227' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/21' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/27' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/31' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/36' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/20' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/39' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/34' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/38' 2024-03-19T02:58:48.568 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/28' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/24' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/29' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/32' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/22' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/25' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/30' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/33' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/23' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/35' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/26' 2024-03-19T02:58:48.569 INFO:tasks.workunit.client.0.smithi092.stdout:removed './228/37' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './228' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/21' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/27' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/31' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/36' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/20' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/39' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/34' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/38' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/28' 2024-03-19T02:58:48.570 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/24' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/29' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/32' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/22' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/25' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/30' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/33' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/23' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/35' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/26' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './229/37' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './229' 2024-03-19T02:58:48.571 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/21' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/27' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/31' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/36' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/20' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/39' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/34' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/38' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/28' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/24' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/29' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/32' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/22' 2024-03-19T02:58:48.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/25' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/30' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/33' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/23' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/35' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/26' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './230/37' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './230' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/21' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/27' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/31' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/36' 2024-03-19T02:58:48.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/20' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/39' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/34' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/38' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/28' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/24' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/29' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/32' 2024-03-19T02:58:48.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/22' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/25' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/30' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/33' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/23' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/35' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/26' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './231/37' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './231' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/21' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/27' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/31' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/36' 2024-03-19T02:58:48.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/20' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/39' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/34' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/38' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/28' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/24' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/29' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/32' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/22' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/25' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/30' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/33' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/23' 2024-03-19T02:58:48.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/35' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/26' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './232/37' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './232' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/21' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/27' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/31' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/36' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/20' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/39' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/34' 2024-03-19T02:58:48.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/38' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/28' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/24' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/29' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/32' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/22' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/25' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/30' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/33' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/23' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/35' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/26' 2024-03-19T02:58:48.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './233/37' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './233' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/21' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/27' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/31' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/36' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/20' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/39' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/34' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/38' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/28' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/24' 2024-03-19T02:58:48.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/29' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/32' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/22' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/25' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/30' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/33' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/23' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/35' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/26' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './234/37' 2024-03-19T02:58:48.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './234' 2024-03-19T02:58:48.581 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/21' 2024-03-19T02:58:48.581 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/27' 2024-03-19T02:58:48.581 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/31' 2024-03-19T02:58:48.581 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/36' 2024-03-19T02:58:48.581 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/20' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/39' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/34' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/38' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/28' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/24' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/29' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/32' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/22' 2024-03-19T02:58:48.582 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/25' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/30' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/33' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/23' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/35' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/26' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './235/37' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './235' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/21' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/27' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/31' 2024-03-19T02:58:48.583 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/36' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/20' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/39' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/34' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/38' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/28' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/24' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/29' 2024-03-19T02:58:48.584 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/32' 2024-03-19T02:58:49.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/22' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/25' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/30' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/33' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/23' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/35' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/26' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './236/37' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './236' 2024-03-19T02:58:49.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/21' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/27' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/31' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/36' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/20' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/39' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/34' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/38' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/28' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/24' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/29' 2024-03-19T02:58:49.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/32' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/22' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/25' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/30' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/33' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/23' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/35' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/26' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './237/37' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './237' 2024-03-19T02:58:49.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/21' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/27' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/31' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/36' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/20' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/39' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/34' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/38' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/28' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/24' 2024-03-19T02:58:49.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/29' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/32' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/22' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/25' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/30' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/33' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/23' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/35' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/26' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './238/37' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './238' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/21' 2024-03-19T02:58:49.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/27' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/31' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/36' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/20' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/39' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/34' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/38' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/28' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/24' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/29' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/32' 2024-03-19T02:58:49.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/22' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/25' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/30' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/33' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/23' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/35' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/26' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './239/37' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './239' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/21' 2024-03-19T02:58:49.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/27' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/31' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/36' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/20' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/39' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/34' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/38' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/28' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/24' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/29' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/32' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/22' 2024-03-19T02:58:49.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/25' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/30' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/33' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/23' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/35' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/26' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './240/37' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './240' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/21' 2024-03-19T02:58:49.905 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/27' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/31' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/36' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/20' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/39' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/34' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/38' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/28' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/24' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/29' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/32' 2024-03-19T02:58:49.906 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/22' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/25' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/30' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/33' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/23' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/35' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/26' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './241/37' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './241' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/21' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/27' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/31' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/36' 2024-03-19T02:58:49.907 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/20' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/39' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/34' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/38' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/28' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/24' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/29' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/32' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/22' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/25' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/30' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/33' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/23' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/35' 2024-03-19T02:58:49.908 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/26' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './242/37' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './242' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/21' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/27' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/31' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/36' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/20' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/39' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/34' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/38' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/28' 2024-03-19T02:58:49.909 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/24' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/29' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/32' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/22' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/25' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/30' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/33' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/23' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/35' 2024-03-19T02:58:49.910 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/26' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './243/37' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './243' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/21' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/27' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/31' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/36' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/20' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/39' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/34' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/38' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/28' 2024-03-19T02:58:49.911 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/24' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/29' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/32' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/22' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/25' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/30' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/33' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/23' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/35' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/26' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './244/37' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './244' 2024-03-19T02:58:49.912 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/21' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/27' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/31' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/36' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/20' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/39' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/34' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/38' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/28' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/24' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/29' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/32' 2024-03-19T02:58:49.913 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/22' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/25' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/30' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/33' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/23' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/35' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/26' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './245/37' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './245' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/21' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/27' 2024-03-19T02:58:49.914 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/31' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/36' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/20' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/39' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/34' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/38' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/28' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/24' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/29' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/32' 2024-03-19T02:58:49.915 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/22' 2024-03-19T02:58:52.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/25' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/30' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/33' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/23' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/35' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/26' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './246/37' 2024-03-19T02:58:52.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './246' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/21' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/27' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/31' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/36' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/20' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/39' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/34' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/38' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/28' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/24' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/29' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/32' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/22' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/25' 2024-03-19T02:58:52.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/30' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/33' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/23' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/35' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/26' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './247/37' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './247' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/21' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/27' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/31' 2024-03-19T02:58:52.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/36' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/20' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/39' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/34' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/38' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/28' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/24' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/29' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/32' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/22' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/25' 2024-03-19T02:58:52.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/30' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/33' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/23' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/35' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/26' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './248/37' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './248' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/21' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/27' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/31' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/36' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/20' 2024-03-19T02:58:52.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/39' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/34' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/38' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/28' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/24' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/29' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/32' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/22' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/25' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/30' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/33' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/23' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/35' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/26' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './249/37' 2024-03-19T02:58:52.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './249' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/21' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/27' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/31' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/36' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/20' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/39' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/34' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/38' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/28' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/24' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/29' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/32' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/22' 2024-03-19T02:58:52.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/25' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/30' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/33' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/23' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/35' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/26' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './250/37' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './250' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/21' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/27' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/31' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/36' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/20' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/39' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/34' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/38' 2024-03-19T02:58:52.363 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/28' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/24' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/29' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/32' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/22' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/25' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/30' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/33' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/23' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/35' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/26' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './251/37' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './251' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/21' 2024-03-19T02:58:52.364 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/27' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/31' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/36' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/20' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/39' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/34' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/38' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/28' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/24' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/29' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/32' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/22' 2024-03-19T02:58:52.365 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/25' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/30' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/33' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/23' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/35' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/26' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './252/37' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './252' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/21' 2024-03-19T02:58:52.366 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/27' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/31' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/36' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/20' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/39' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/34' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/38' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/28' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/24' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/29' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/32' 2024-03-19T02:58:52.367 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/22' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/25' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/30' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/33' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/23' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/35' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/26' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './253/37' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './253' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/21' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/27' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/31' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/36' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/20' 2024-03-19T02:58:52.368 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/39' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/34' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/38' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/28' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/24' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/29' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/32' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/22' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/25' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/30' 2024-03-19T02:58:52.369 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/33' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/23' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/35' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/26' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './254/37' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './254' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/21' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/27' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/31' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/36' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/20' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/39' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/34' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/38' 2024-03-19T02:58:52.370 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/28' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/24' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/29' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/32' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/22' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/25' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/30' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/33' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/23' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/35' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/26' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './255/37' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './255' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/21' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/27' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/31' 2024-03-19T02:58:52.371 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/36' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/20' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/39' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/34' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/38' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/28' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/24' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/29' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/32' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/22' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/25' 2024-03-19T02:58:52.372 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/30' 2024-03-19T02:58:53.141 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/33' 2024-03-19T02:58:53.141 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/23' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/35' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/26' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './256/37' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './256' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/21' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/27' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/31' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/36' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/20' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/39' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/34' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/38' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/28' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/24' 2024-03-19T02:58:53.142 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/29' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/32' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/22' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/25' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/30' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/33' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/23' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/35' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/26' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './257/37' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './257' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/21' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/27' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/31' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/36' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/20' 2024-03-19T02:58:53.143 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/39' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/34' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/38' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/28' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/24' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/29' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/32' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/22' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/25' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/30' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/33' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/23' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/35' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/26' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed './258/37' 2024-03-19T02:58:53.144 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './258' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/21' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/27' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/31' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/36' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/20' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/39' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/34' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/38' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/28' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/24' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/29' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/32' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/22' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/25' 2024-03-19T02:58:53.145 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/30' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/33' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/23' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/35' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/26' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './259/37' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './259' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/21' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/27' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/31' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/36' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/20' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/39' 2024-03-19T02:58:53.146 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/34' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/38' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/28' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/24' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/29' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/32' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/22' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/25' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/30' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/33' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/23' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/35' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/26' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './260/37' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './260' 2024-03-19T02:58:53.147 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/21' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/27' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/31' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/36' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/20' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/39' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/34' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/38' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/28' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/24' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/29' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/32' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/22' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/25' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/30' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/33' 2024-03-19T02:58:53.148 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/23' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/35' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/26' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './261/37' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './261' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/21' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/27' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/31' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/36' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/20' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/39' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/34' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/38' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/28' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/24' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/29' 2024-03-19T02:58:53.149 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/32' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/22' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/25' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/30' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/33' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/23' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/35' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/26' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './262/37' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './262' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/21' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/27' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/31' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/36' 2024-03-19T02:58:53.150 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/20' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/39' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/34' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/38' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/28' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/24' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/29' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/32' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/22' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/25' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/30' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/33' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/23' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/35' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/26' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './263/37' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './263' 2024-03-19T02:58:53.151 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/21' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/27' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/31' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/36' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/20' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/39' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/34' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/38' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/28' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/24' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/29' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/32' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/22' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/25' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/30' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/33' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/23' 2024-03-19T02:58:53.152 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/35' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/26' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './264/37' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './264' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/21' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/27' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/31' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/36' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/20' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/39' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/34' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/38' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/28' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/24' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/29' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/32' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/22' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/25' 2024-03-19T02:58:53.153 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/30' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/33' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/23' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/35' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/26' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './265/37' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './265' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/21' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/27' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/31' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/36' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/20' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/39' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/34' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/38' 2024-03-19T02:58:53.154 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/28' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/24' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/29' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/32' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/22' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/25' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/30' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/33' 2024-03-19T02:58:53.155 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/23' 2024-03-19T02:58:54.265 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/35' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/26' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './266/37' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './266' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/21' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/27' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/31' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/36' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/20' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/39' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/34' 2024-03-19T02:58:54.266 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/38' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/28' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/24' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/29' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/32' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/22' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/25' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/30' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/33' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/23' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/35' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/26' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './267/37' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './267' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/21' 2024-03-19T02:58:54.267 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/27' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/31' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/36' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/20' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/39' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/34' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/38' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/28' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/24' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/29' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/32' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/22' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/25' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/30' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/33' 2024-03-19T02:58:54.268 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/23' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/35' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/26' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './268/37' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './268' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/21' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/27' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/31' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/36' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/20' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/39' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/34' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/38' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/28' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/24' 2024-03-19T02:58:54.269 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/29' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/32' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/22' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/25' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/30' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/33' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/23' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/35' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/26' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './269/37' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './269' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/21' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/27' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/31' 2024-03-19T02:58:54.270 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/36' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/20' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/39' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/34' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/38' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/28' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/24' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/29' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/32' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/22' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/25' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/30' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/33' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/23' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/35' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/26' 2024-03-19T02:58:54.271 INFO:tasks.workunit.client.0.smithi092.stdout:removed './270/37' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './270' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/21' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/27' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/31' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/36' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/20' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/39' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/34' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/38' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/28' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/24' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/29' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/32' 2024-03-19T02:58:54.272 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/22' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/25' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/30' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/33' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/23' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/35' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/26' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './271/37' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './271' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/21' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/27' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/31' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/36' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/20' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/39' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/34' 2024-03-19T02:58:54.273 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/38' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/28' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/24' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/29' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/32' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/22' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/25' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/30' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/33' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/23' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/35' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/26' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './272/37' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './272' 2024-03-19T02:58:54.274 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/21' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/27' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/31' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/36' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/20' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/39' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/34' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/38' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/28' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/24' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/29' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/32' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/22' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/25' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/30' 2024-03-19T02:58:54.275 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/33' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/23' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/35' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/26' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './273/37' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './273' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/21' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/27' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/31' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/36' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/20' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/39' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/34' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/38' 2024-03-19T02:58:54.276 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/28' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/24' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/29' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/32' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/22' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/25' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/30' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/33' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/23' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/35' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/26' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './274/37' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './274' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/21' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/27' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/31' 2024-03-19T02:58:54.277 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/36' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/20' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/39' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/34' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/38' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/28' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/24' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/29' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/32' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/22' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/25' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/30' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/33' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/23' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/35' 2024-03-19T02:58:54.278 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/26' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './275/37' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './275' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/21' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/27' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/31' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/36' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/20' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/39' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/34' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/38' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/28' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/24' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/29' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/32' 2024-03-19T02:58:54.279 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/22' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/25' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/30' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/33' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/23' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/35' 2024-03-19T02:58:54.280 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/26' 2024-03-19T02:58:56.347 INFO:tasks.workunit.client.0.smithi092.stdout:removed './276/37' 2024-03-19T02:58:56.347 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './276' 2024-03-19T02:58:56.347 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/21' 2024-03-19T02:58:56.347 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/27' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/31' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/36' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/20' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/39' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/34' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/38' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/28' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/24' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/29' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/32' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/22' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/25' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/30' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/33' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/23' 2024-03-19T02:58:56.348 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/35' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/26' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './277/37' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './277' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/21' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/27' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/31' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/36' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/20' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/39' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/34' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/38' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/28' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/24' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/29' 2024-03-19T02:58:56.349 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/32' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/22' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/25' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/30' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/33' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/23' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/35' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/26' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './278/37' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './278' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/21' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/27' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/31' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/36' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/20' 2024-03-19T02:58:56.350 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/39' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/34' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/38' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/28' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/24' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/29' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/32' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/22' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/25' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/30' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/33' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/23' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/35' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/26' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed './279/37' 2024-03-19T02:58:56.351 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './279' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/21' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/27' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/31' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/36' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/20' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/39' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/34' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/38' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/28' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/24' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/29' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/32' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/22' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/25' 2024-03-19T02:58:56.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/30' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/33' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/23' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/35' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/26' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './280/37' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './280' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/21' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/27' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/31' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/36' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/20' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/39' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/34' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/38' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/28' 2024-03-19T02:58:56.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/24' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/29' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/32' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/22' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/25' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/30' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/33' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/23' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/35' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/26' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './281/37' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './281' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/21' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/27' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/31' 2024-03-19T02:58:56.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/36' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/20' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/39' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/34' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/38' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/28' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/24' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/29' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/32' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/22' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/25' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/30' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/33' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/23' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/35' 2024-03-19T02:58:56.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/26' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './282/37' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './282' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/21' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/27' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/31' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/36' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/20' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/39' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/34' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/38' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/28' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/24' 2024-03-19T02:58:56.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/29' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/32' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/22' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/25' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/30' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/33' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/23' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/35' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/26' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './283/37' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './283' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/21' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/27' 2024-03-19T02:58:56.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/31' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/36' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/20' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/39' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/34' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/38' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/28' 2024-03-19T02:58:56.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/24' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/29' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/32' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/22' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/25' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/30' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/33' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/23' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/35' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/26' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './284/37' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './284' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/21' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/27' 2024-03-19T02:58:56.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/31' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/36' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/20' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/39' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/34' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/38' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/28' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/24' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/29' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/32' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/22' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/25' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/30' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/33' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/23' 2024-03-19T02:58:56.360 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/35' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/26' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './285/37' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './285' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/21' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/27' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/31' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/36' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/20' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/39' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/34' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/38' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/28' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/24' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/29' 2024-03-19T02:58:56.361 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/32' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/22' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/25' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/30' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/33' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/23' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/35' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/26' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed './286/37' 2024-03-19T02:58:56.362 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './286' 2024-03-19T02:58:57.888 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/21' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/27' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/31' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/36' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/20' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/39' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/34' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/38' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/28' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/24' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/29' 2024-03-19T02:58:57.889 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/32' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/22' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/25' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/30' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/33' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/23' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/35' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/26' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './287/37' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './287' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/21' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/27' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/31' 2024-03-19T02:58:57.890 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/36' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/20' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/39' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/34' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/38' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/28' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/24' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/29' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/32' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/22' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/25' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/30' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/33' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/23' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/35' 2024-03-19T02:58:57.891 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/26' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './288/37' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './288' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/21' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/27' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/31' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/36' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/20' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/39' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/34' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/38' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/28' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/24' 2024-03-19T02:58:57.892 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/29' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/32' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/22' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/25' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/30' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/33' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/23' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/35' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/26' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './289/37' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './289' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/21' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/27' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/31' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/36' 2024-03-19T02:58:57.893 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/20' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/39' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/34' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/38' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/28' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/24' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/29' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/32' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/22' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/25' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/30' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/33' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/23' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/35' 2024-03-19T02:58:57.894 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/26' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './290/37' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './290' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/21' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/27' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/31' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/36' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/20' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/39' 2024-03-19T02:58:57.895 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/34' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/38' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/28' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/24' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/29' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/32' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/22' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/25' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/30' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/33' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/23' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/35' 2024-03-19T02:58:57.896 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/26' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './291/37' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './291' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/21' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/27' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/31' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/36' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/20' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/39' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/34' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/38' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/28' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/24' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/29' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/32' 2024-03-19T02:58:57.897 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/22' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/25' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/30' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/33' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/23' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/35' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/26' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './292/37' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './292' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/21' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/27' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/31' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/36' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/20' 2024-03-19T02:58:57.898 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/39' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/34' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/38' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/28' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/24' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/29' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/32' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/22' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/25' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/30' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/33' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/23' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/35' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/26' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed './293/37' 2024-03-19T02:58:57.899 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './293' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/21' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/27' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/31' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/36' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/20' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/39' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/34' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/38' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/28' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/24' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/29' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/32' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/22' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/25' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/30' 2024-03-19T02:58:57.900 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/33' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/23' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/35' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/26' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './294/37' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './294' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/21' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/27' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/31' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/36' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/20' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/39' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/34' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/38' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/28' 2024-03-19T02:58:57.901 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/24' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/29' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/32' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/22' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/25' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/30' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/33' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/23' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/35' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/26' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './295/37' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './295' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/21' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/27' 2024-03-19T02:58:57.902 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/31' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/36' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/20' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/39' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/34' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/38' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/28' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/24' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/29' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/32' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/22' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/25' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/30' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/33' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/23' 2024-03-19T02:58:57.903 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/35' 2024-03-19T02:58:57.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/26' 2024-03-19T02:58:57.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './296/37' 2024-03-19T02:58:57.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './296' 2024-03-19T02:58:57.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/21' 2024-03-19T02:58:57.904 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/27' 2024-03-19T02:58:59.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:58:59.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/31' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/36' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/20' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/39' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/34' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/38' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/28' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/24' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/29' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/32' 2024-03-19T02:59:00.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/22' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/25' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/30' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/33' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/23' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/35' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/26' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './297/37' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './297' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/21' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/27' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/31' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/36' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/20' 2024-03-19T02:59:00.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/39' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/34' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/38' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/28' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/24' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/29' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/32' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/22' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/25' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/30' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/33' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/23' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/35' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/26' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './298/37' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './298' 2024-03-19T02:59:00.423 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/21' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/27' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/31' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/36' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/20' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/39' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/34' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/38' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/28' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/24' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/29' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/32' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/22' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/25' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/30' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/33' 2024-03-19T02:59:00.424 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/23' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/35' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/26' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './299/37' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './299' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/19' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/15' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/20' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/12' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/10' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/11' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/16' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/14' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/18' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/13' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/166/17' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/166' 2024-03-19T02:59:00.425 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/19' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/15' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/20' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/12' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/10' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/11' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/16' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/14' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/18' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/13' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/147/17' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/147' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/19' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/15' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/20' 2024-03-19T02:59:00.426 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/12' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/10' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/11' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/16' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/14' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/18' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/13' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/151/17' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/151' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/19' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/15' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/20' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/12' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/10' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/11' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/16' 2024-03-19T02:59:00.427 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/14' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/18' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/13' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/170/17' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/170' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/19' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/15' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/20' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/12' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/10' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/11' 2024-03-19T02:59:00.428 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/16' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/14' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/18' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/13' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/167/17' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/167' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/19' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/15' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/20' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/12' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/10' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/11' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/16' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/14' 2024-03-19T02:59:00.429 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/18' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/13' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/140/17' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/140' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/19' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/15' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/20' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/12' 2024-03-19T02:59:00.430 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/10' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/11' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/16' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/14' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/18' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/13' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/142/17' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/142' 2024-03-19T02:59:02.731 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/19' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/15' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/20' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/12' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/10' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/11' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/16' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/14' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/18' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/13' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/150/17' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/150' 2024-03-19T02:59:02.732 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/19' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/15' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/20' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/12' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/10' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/11' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/16' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/14' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/18' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/13' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/105/17' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/105' 2024-03-19T02:59:02.733 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/19' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/15' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/20' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/12' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/10' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/11' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/16' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/14' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/18' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/13' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/149/17' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/149' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/19' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/15' 2024-03-19T02:59:02.734 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/20' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/12' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/10' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/11' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/16' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/14' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/18' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/13' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/109/17' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/109' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/19' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/15' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/20' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/12' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/10' 2024-03-19T02:59:02.735 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/11' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/16' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/14' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/18' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/13' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/158/17' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/158' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/19' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/15' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/20' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/12' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/10' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/11' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/16' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/14' 2024-03-19T02:59:02.736 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/18' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/13' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/106/17' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/106' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/19' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/15' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/20' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/12' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/10' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/11' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/16' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/14' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/18' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/13' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/189/17' 2024-03-19T02:59:02.737 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/189' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/19' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/15' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/20' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/12' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/10' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/11' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/16' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/14' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/18' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/13' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/133/17' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/133' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/19' 2024-03-19T02:59:02.738 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/15' 2024-03-19T02:59:02.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/20' 2024-03-19T02:59:02.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/12' 2024-03-19T02:59:02.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/10' 2024-03-19T02:59:02.739 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/11' 2024-03-19T02:59:03.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/16' 2024-03-19T02:59:03.081 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/14' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/18' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/13' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/173/17' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/173' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/19' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/15' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/20' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/12' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/10' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/11' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/16' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/14' 2024-03-19T02:59:03.082 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/18' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/13' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/174/17' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/174' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/19' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/15' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/20' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/12' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/10' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/11' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/16' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/14' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/18' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/13' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/155/17' 2024-03-19T02:59:03.083 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/155' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/19' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/15' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/20' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/12' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/10' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/11' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/16' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/14' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/18' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/13' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/114/17' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/114' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/19' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/15' 2024-03-19T02:59:03.084 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/20' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/12' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/10' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/11' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/16' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/14' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/18' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/13' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/159/17' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/159' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/19' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/15' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/20' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/12' 2024-03-19T02:59:03.085 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/10' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/11' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/16' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/14' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/18' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/13' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/187/17' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/187' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/19' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/15' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/20' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/12' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/10' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/11' 2024-03-19T02:59:03.086 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/16' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/14' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/18' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/13' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/185/17' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/185' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/19' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/15' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/20' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/12' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/10' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/11' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/16' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/14' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/18' 2024-03-19T02:59:03.087 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/13' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/120/17' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/120' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/19' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/15' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/20' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/12' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/10' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/11' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/16' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/14' 2024-03-19T02:59:03.088 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/18' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/13' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/110/17' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/110' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/19' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/15' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/20' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/12' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/10' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/11' 2024-03-19T02:59:03.089 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/16' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/14' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/18' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/13' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/137/17' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/137' 2024-03-19T02:59:03.415 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/19' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/15' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/20' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/12' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/10' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/11' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/16' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/14' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/18' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/13' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/198/17' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/198' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/19' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/15' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/20' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/12' 2024-03-19T02:59:03.416 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/10' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/11' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/16' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/14' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/18' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/13' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/136/17' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/136' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/19' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/15' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/20' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/12' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/10' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/11' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/16' 2024-03-19T02:59:03.417 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/14' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/18' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/13' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/125/17' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/125' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/19' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/15' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/20' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/12' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/10' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/11' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/16' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/14' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/18' 2024-03-19T02:59:03.418 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/13' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/194/17' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/194' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/19' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/15' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/20' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/12' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/10' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/11' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/16' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/14' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/18' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/13' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/119/17' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/119' 2024-03-19T02:59:03.419 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/19' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/15' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/20' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/12' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/10' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/11' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/16' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/14' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/18' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/13' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/157/17' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/157' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/19' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/15' 2024-03-19T02:59:03.420 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/20' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/12' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/10' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/11' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/16' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/14' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/18' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/13' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/172/17' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/172' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/19' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/15' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/20' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/12' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/10' 2024-03-19T02:59:03.421 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/11' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/16' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/14' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/18' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/13' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/186/17' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/186' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/19' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/15' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/20' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/12' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/10' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/11' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/16' 2024-03-19T02:59:03.422 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/14' 2024-03-19T02:59:03.770 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/18' 2024-03-19T02:59:03.770 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/13' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/190/17' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/190' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/19' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/15' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/20' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/12' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/10' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/11' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/16' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/14' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/18' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/13' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/104/17' 2024-03-19T02:59:03.771 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/104' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/19' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/15' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/20' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/12' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/10' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/11' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/16' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/14' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/18' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/13' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/164/17' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/164' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/19' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/15' 2024-03-19T02:59:03.772 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/20' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/12' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/10' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/11' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/16' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/14' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/18' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/13' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/112/17' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/112' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/19' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/15' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/20' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/12' 2024-03-19T02:59:03.773 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/10' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/11' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/16' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/14' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/18' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/13' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/100/17' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/100' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/19' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/15' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/20' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/12' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/10' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/11' 2024-03-19T02:59:03.774 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/16' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/14' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/18' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/13' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/134/17' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/134' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/19' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/15' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/20' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/12' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/10' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/11' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/16' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/14' 2024-03-19T02:59:03.775 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/18' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/13' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/115/17' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/115' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/19' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/15' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/20' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/12' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/10' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/11' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/16' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/14' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/18' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/13' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/116/17' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/116' 2024-03-19T02:59:03.776 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/19' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/15' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/20' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/12' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/10' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/11' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/16' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/14' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/18' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/13' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/195/17' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/195' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/19' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/15' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/20' 2024-03-19T02:59:03.777 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/12' 2024-03-19T02:59:03.778 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/10' 2024-03-19T02:59:03.778 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/11' 2024-03-19T02:59:03.778 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/16' 2024-03-19T02:59:03.778 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/14' 2024-03-19T02:59:03.778 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/18' 2024-03-19T02:59:04.123 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/13' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/138/17' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/138' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/19' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/15' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/20' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/12' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/10' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/11' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/16' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/14' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/18' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/13' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/135/17' 2024-03-19T02:59:04.124 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/135' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/19' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/15' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/20' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/12' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/10' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/11' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/16' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/14' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/18' 2024-03-19T02:59:04.125 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/13' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/117/17' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/117' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/19' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/15' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/20' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/12' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/10' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/11' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/16' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/14' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/18' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/13' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/143/17' 2024-03-19T02:59:04.126 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/143' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/19' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/15' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/20' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/12' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/10' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/11' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/16' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/14' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/18' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/13' 2024-03-19T02:59:04.127 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/180/17' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/180' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/19' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/15' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/20' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/12' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/10' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/11' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/16' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/14' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/18' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/13' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/126/17' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/126' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/19' 2024-03-19T02:59:04.128 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/15' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/20' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/12' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/10' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/11' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/16' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/14' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/18' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/13' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/161/17' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/161' 2024-03-19T02:59:04.129 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/19' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/15' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/20' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/12' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/10' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/11' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/16' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/14' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/18' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/13' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/176/17' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/176' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/19' 2024-03-19T02:59:04.130 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/15' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/20' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/12' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/10' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/11' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/16' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/14' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/18' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/13' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/111/17' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/111' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/19' 2024-03-19T02:59:04.131 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/15' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/20' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/12' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/10' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/11' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/16' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/14' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/18' 2024-03-19T02:59:04.132 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/13' 2024-03-19T02:59:05.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/191/17' 2024-03-19T02:59:05.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/191' 2024-03-19T02:59:05.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/19' 2024-03-19T02:59:05.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/15' 2024-03-19T02:59:05.572 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/20' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/12' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/10' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/11' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/16' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/14' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/18' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/13' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/184/17' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/184' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/19' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/15' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/20' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/12' 2024-03-19T02:59:05.573 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/10' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/11' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/16' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/14' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/18' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/13' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/192/17' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/192' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/19' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/15' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/20' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/12' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/10' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/11' 2024-03-19T02:59:05.574 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/16' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/14' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/18' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/13' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/103/17' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/103' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/19' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/15' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/20' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/12' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/10' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/11' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/16' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/14' 2024-03-19T02:59:05.575 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/18' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/13' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/128/17' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/128' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/19' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/15' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/20' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/12' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/10' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/11' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/16' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/14' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/18' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/13' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/144/17' 2024-03-19T02:59:05.576 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/144' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/19' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/15' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/20' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/12' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/10' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/11' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/16' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/14' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/18' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/13' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/146/17' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/146' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/19' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/15' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/20' 2024-03-19T02:59:05.577 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/12' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/10' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/11' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/16' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/14' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/18' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/13' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/141/17' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/141' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/19' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/15' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/20' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/12' 2024-03-19T02:59:05.578 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/10' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/11' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/16' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/14' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/18' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/13' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/182/17' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/182' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/19' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/15' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/20' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/12' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/10' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/11' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/16' 2024-03-19T02:59:05.579 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/14' 2024-03-19T02:59:05.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/18' 2024-03-19T02:59:05.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/13' 2024-03-19T02:59:05.580 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/107/17' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/107' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/19' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/15' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/20' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/12' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/10' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/11' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/16' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/14' 2024-03-19T02:59:06.788 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/18' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/13' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/178/17' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/178' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/19' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/15' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/20' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/12' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/10' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/11' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/16' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/14' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/18' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/13' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/156/17' 2024-03-19T02:59:06.789 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/156' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/19' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/15' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/20' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/12' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/10' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/11' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/16' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/14' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/18' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/13' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/131/17' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/131' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/19' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/15' 2024-03-19T02:59:06.790 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/20' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/12' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/10' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/11' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/16' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/14' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/18' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/13' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/123/17' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/123' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/19' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/15' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/20' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/12' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/10' 2024-03-19T02:59:06.791 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/11' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/16' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/14' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/18' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/13' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/160/17' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/160' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/19' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/15' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/20' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/12' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/10' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/11' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/16' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/14' 2024-03-19T02:59:06.792 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/18' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/13' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/165/17' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/165' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/19' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/15' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/20' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/12' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/10' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/11' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/16' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/14' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/18' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/13' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/101/17' 2024-03-19T02:59:06.793 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/101' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/19' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/15' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/20' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/12' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/10' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/11' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/16' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/14' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/18' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/13' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/148/17' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/148' 2024-03-19T02:59:06.794 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/19' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/15' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/20' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/12' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/10' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/11' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/16' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/14' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/18' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/13' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/168/17' 2024-03-19T02:59:06.795 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/168' 2024-03-19T02:59:08.058 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/19' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/15' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/20' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/12' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/10' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/11' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/16' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/14' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/18' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/13' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/179/17' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/179' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/19' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/15' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/20' 2024-03-19T02:59:08.059 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/12' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/10' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/11' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/16' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/14' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/18' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/13' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/132/17' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/132' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/19' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/15' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/20' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/12' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/10' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/11' 2024-03-19T02:59:08.060 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/16' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/14' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/18' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/13' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/199/17' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/199' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/19' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/15' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/20' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/12' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/10' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/11' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/16' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/14' 2024-03-19T02:59:08.061 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/18' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/13' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/163/17' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/163' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/19' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/15' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/20' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/12' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/10' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/11' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/16' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/14' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/18' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/13' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/153/17' 2024-03-19T02:59:08.062 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/153' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/19' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/15' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/20' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/12' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/10' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/11' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/16' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/14' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/18' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/13' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/122/17' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/122' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/19' 2024-03-19T02:59:08.063 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/15' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/20' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/12' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/10' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/11' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/16' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/14' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/18' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/13' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/162/17' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/162' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/19' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/15' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/20' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/12' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/10' 2024-03-19T02:59:08.064 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/11' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/16' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/14' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/18' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/13' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/197/17' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/197' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/19' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/15' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/20' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/12' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/10' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/11' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/16' 2024-03-19T02:59:08.065 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/14' 2024-03-19T02:59:08.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/18' 2024-03-19T02:59:08.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/13' 2024-03-19T02:59:08.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/113/17' 2024-03-19T02:59:08.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/113' 2024-03-19T02:59:08.066 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/19' 2024-03-19T02:59:09.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/15' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/20' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/12' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/10' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/11' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/16' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/14' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/18' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/13' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/152/17' 2024-03-19T02:59:09.352 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/152' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/19' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/15' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/20' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/12' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/10' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/11' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/16' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/14' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/18' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/13' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/121/17' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/121' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/19' 2024-03-19T02:59:09.353 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/15' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/20' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/12' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/10' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/11' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/16' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/14' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/18' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/13' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/118/17' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/118' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/19' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/15' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/20' 2024-03-19T02:59:09.354 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/12' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/10' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/11' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/16' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/14' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/18' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/13' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/196/17' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/196' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/19' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/15' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/20' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/12' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/10' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/11' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/16' 2024-03-19T02:59:09.355 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/14' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/18' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/13' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/124/17' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/124' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/19' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/15' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/20' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/12' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/10' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/11' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/16' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/14' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/18' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/13' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/188/17' 2024-03-19T02:59:09.356 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/188' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/19' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/15' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/20' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/12' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/10' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/11' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/16' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/14' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/18' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/13' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/145/17' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/145' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/19' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/15' 2024-03-19T02:59:09.357 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/20' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/12' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/10' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/11' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/16' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/14' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/18' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/13' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/130/17' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/130' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/19' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/15' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/20' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/12' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/10' 2024-03-19T02:59:09.358 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/11' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/16' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/14' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/18' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/13' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/193/17' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/193' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/19' 2024-03-19T02:59:09.359 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/15' 2024-03-19T02:59:09.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.463+0000 7f0337200640 1 Processor -- start 2024-03-19T02:59:09.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.464+0000 7f0337200640 1 -- start start 2024-03-19T02:59:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0338154c20 0x7f033815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03380c8dc0 0x7f03380cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f033815ae10 con 0x7f03380c8dc0 2024-03-19T02:59:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f03380c9ef0 con 0x7f0338154c20 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0337200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f03380ca190 con 0x7f0338155000 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0335e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0338154c20 0x7f033815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0336800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0335e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0338154c20 0x7f033815b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:50976/0 (socket says 172.21.15.92:50976) 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0336800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46742/0 (socket says 172.21.15.92:46742) 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.465+0000 7f0335e00640 1 -- 172.21.15.92:0/232390335 learned_addr learned my addr 172.21.15.92:0/232390335 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f032d400640 1 --2- 172.21.15.92:0/232390335 >> v2:172.21.15.92:3300/0 conn(0x7f03380c8dc0 0x7f03380cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0335e00640 1 -- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 msgr2=0x7f03380cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0335e00640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0335e00640 1 -- 172.21.15.92:0/232390335 >> v2:172.21.15.92:3300/0 conn(0x7f03380c8dc0 msgr2=0x7f03380cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0335e00640 1 --2- 172.21.15.92:0/232390335 >> v2:172.21.15.92:3300/0 conn(0x7f03380c8dc0 0x7f03380cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0335e00640 1 -- 172.21.15.92:0/232390335 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03380ca450 con 0x7f0338154c20 2024-03-19T02:59:09.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.466+0000 7f0336800640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.467+0000 7f0335e00640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338154c20 0x7f033815b2a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f03380cae20 tx=0x7f032802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d46d06c3320248c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.467+0000 7f0335400640 1 -- 172.21.15.92:0/232390335 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032802cb00 con 0x7f0338154c20 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0335400640 1 -- 172.21.15.92:0/232390335 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f032802cca0 con 0x7f0338154c20 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0335400640 1 -- 172.21.15.92:0/232390335 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0328034610 con 0x7f0338154c20 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 -- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338154c20 msgr2=0x7f033815b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338154c20 0x7f033815b2a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f03380cae20 tx=0x7f032802bf60 comp rx=0 tx=0).stop 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 -- 172.21.15.92:0/232390335 shutdown_connections 2024-03-19T02:59:09.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338155000 0x7f03380cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 --2- 172.21.15.92:0/232390335 >> v2:172.21.15.92:3300/0 conn(0x7f03380c8dc0 0x7f03380cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 --2- 172.21.15.92:0/232390335 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338154c20 0x7f033815b2a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 -- 172.21.15.92:0/232390335 >> 172.21.15.92:0/232390335 conn(0x7f0338070d10 msgr2=0x7f0338071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 -- 172.21.15.92:0/232390335 shutdown_connections 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.468+0000 7f0337200640 1 -- 172.21.15.92:0/232390335 wait complete. 2024-03-19T02:59:09.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.470+0000 7f0337200640 1 Processor -- start 2024-03-19T02:59:09.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.470+0000 7f0337200640 1 -- start start 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03380c8dc0 0x7f03380b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338155000 0x7f03381f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f03380c47d0 con 0x7f03380c8dc0 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f03380caa10 con 0x7f0338155000 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0337200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f03380c4950 con 0x7f0338154c20 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f032d400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0335e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03380c8dc0 0x7f03380b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f032d400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46754/0 (socket says 172.21.15.92:46754) 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f032d400640 1 -- 172.21.15.92:0/60004196 learned_addr learned my addr 172.21.15.92:0/60004196 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.471+0000 7f0336800640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338155000 0x7f03381f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 -- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338155000 msgr2=0x7f03381f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338155000 0x7f03381f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 -- 172.21.15.92:0/60004196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03380c8dc0 msgr2=0x7f03380b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03380c8dc0 0x7f03380b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 -- 172.21.15.92:0/60004196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03380c7220 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032d400640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f032000e8a0 tx=0x7f032000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320017070 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f032000ee90 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03380c5960 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032000cb80 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.472+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03380c5f20 con 0x7f0338154c20 2024-03-19T02:59:09.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.474+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f033804f580 con 0x7f0338154c20 2024-03-19T02:59:09.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.474+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f032001a070 con 0x7f0338154c20 2024-03-19T02:59:09.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.475+0000 7f032ea00640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 0x7f031406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.475+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f032008aa10 con 0x7f0338154c20 2024-03-19T02:59:09.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.475+0000 7f0335e00640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 0x7f031406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.476+0000 7f0335e00640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 0x7f031406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0328035000 tx=0x7f0328030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.479+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0320058ca0 con 0x7f0338154c20 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/20' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/12' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/10' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/11' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/16' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/14' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/18' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/13' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/175/17' 2024-03-19T02:59:09.615 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/175' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/19' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/15' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/20' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/12' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/10' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/11' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/16' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/14' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/18' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/13' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/181/17' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/181' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/19' 2024-03-19T02:59:09.616 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/15' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/20' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/12' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/10' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/11' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/16' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/14' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/18' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/13' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/127/17' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/127' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/19' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/15' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/20' 2024-03-19T02:59:09.617 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/12' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/10' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/11' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/16' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/14' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/18' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/13' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/108/17' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/108' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/19' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/15' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/20' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/12' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/10' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/11' 2024-03-19T02:59:09.618 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/16' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/14' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/18' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/13' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/183/17' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/183' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/19' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/15' 2024-03-19T02:59:09.619 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/20' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/12' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/10' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/11' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/16' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/14' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/18' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/13' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/177/17' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/177' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/19' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/15' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/20' 2024-03-19T02:59:09.620 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/12' 2024-03-19T02:59:09.621 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/10' 2024-03-19T02:59:09.621 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/11' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/16' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/14' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/18' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/13' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/169/17' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/169' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/19' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/15' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/20' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/12' 2024-03-19T02:59:09.622 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/10' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/11' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/16' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/14' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/18' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/13' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/154/17' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/154' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/19' 2024-03-19T02:59:09.623 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/15' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/20' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/12' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/10' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/11' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/16' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/14' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/18' 2024-03-19T02:59:09.624 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/13' 2024-03-19T02:59:09.625 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/171/17' 2024-03-19T02:59:09.625 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/171' 2024-03-19T02:59:09.625 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/19' 2024-03-19T02:59:09.625 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/15' 2024-03-19T02:59:09.625 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/20' 2024-03-19T02:59:09.736 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.734+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03380c9650 con 0x7f0338154c20 2024-03-19T02:59:09.737 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.735+0000 7f032ea00640 1 -- 172.21.15.92:0/60004196 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f032005cd40 con 0x7f0338154c20 2024-03-19T02:59:09.737 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:09.737 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:09.737 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 msgr2=0x7f031406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 0x7f031406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0328035000 tx=0x7f0328030000 comp rx=0 tx=0).stop 2024-03-19T02:59:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 msgr2=0x7f03380c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f032000e8a0 tx=0x7f032000eca0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 shutdown_connections 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f031406bbd0 0x7f031406e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0338155000 0x7f03381f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0338154c20 0x7f03380c6ce0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 --2- 172.21.15.92:0/60004196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03380c8dc0 0x7f03380b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.738+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 >> 172.21.15.92:0/60004196 conn(0x7f0338070d10 msgr2=0x7f033806fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.739+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 shutdown_connections 2024-03-19T02:59:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.739+0000 7f0337200640 1 -- 172.21.15.92:0/60004196 wait complete. 2024-03-19T02:59:09.758 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-19T02:59:09.758 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-19T02:59:09.758 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:59:09.770 INFO:tasks.ceph.mds.f:Stopped 2024-03-19T02:59:09.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:09.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.976+0000 7fa3b5a00640 1 Processor -- start 2024-03-19T02:59:09.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.976+0000 7fa3b5a00640 1 -- start start 2024-03-19T02:59:09.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.977+0000 7fa3b5a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa3b00c5480 0x7fa3b0072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.977+0000 7fa3b5a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3b00c5860 0x7fa3b01599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.977+0000 7fa3b5a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b0072140 con 0x7fa3b00c36f0 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3b5a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b0159520 con 0x7fa3b00c5860 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3b5a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b00c4ef0 con 0x7fa3b00c5480 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3afe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa3b00c5480 0x7fa3b0072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3b5000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa3b00c36f0 0x7fa3b0151610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:35506/0 (socket says 172.21.15.92:35506) 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3b5000640 1 -- 172.21.15.92:0/701114807 learned_addr learned my addr 172.21.15.92:0/701114807 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3af400640 1 --2- 172.21.15.92:0/701114807 >> v2:172.21.15.92:3300/0 conn(0x7fa3b00c5860 0x7fa3b01599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3aea00640 1 -- 172.21.15.92:0/701114807 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2400405308 0 0) 0x7fa3b0072140 con 0x7fa3b00c36f0 2024-03-19T02:59:09.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3aea00640 1 -- 172.21.15.92:0/701114807 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa394003520 con 0x7fa3b00c36f0 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3afe00640 1 -- 172.21.15.92:0/701114807 >> v2:172.21.15.92:3300/0 conn(0x7fa3b00c5860 msgr2=0x7fa3b01599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3afe00640 1 --2- 172.21.15.92:0/701114807 >> v2:172.21.15.92:3300/0 conn(0x7fa3b00c5860 0x7fa3b01599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3afe00640 1 -- 172.21.15.92:0/701114807 >> v1:172.21.15.159:6789/0 conn(0x7fa3b00c36f0 legacy=0x7fa3b0151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.978+0000 7fa3afe00640 1 -- 172.21.15.92:0/701114807 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b00728d0 con 0x7fa3b00c5480 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.979+0000 7fa3afe00640 1 --2- 172.21.15.92:0/701114807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c5480 0x7fa3b0072390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa3b01596a0 tx=0x7fa3a002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13da650f7ed87aff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.980+0000 7fa3aea00640 1 -- 172.21.15.92:0/701114807 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00315a0 con 0x7fa3b00c5480 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.980+0000 7fa3aea00640 1 -- 172.21.15.92:0/701114807 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a0037070 con 0x7fa3b00c5480 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.980+0000 7fa3aea00640 1 -- 172.21.15.92:0/701114807 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0031940 con 0x7fa3b00c5480 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.980+0000 7fa3b5a00640 1 -- 172.21.15.92:0/701114807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c5480 msgr2=0x7fa3b0072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.980+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/701114807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c5480 0x7fa3b0072390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa3b01596a0 tx=0x7fa3a002b840 comp rx=0 tx=0).stop 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 -- 172.21.15.92:0/701114807 shutdown_connections 2024-03-19T02:59:09.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/701114807 >> v2:172.21.15.92:3300/0 conn(0x7fa3b00c5860 0x7fa3b01599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/701114807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c5480 0x7fa3b0072390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 -- 172.21.15.92:0/701114807 >> 172.21.15.92:0/701114807 conn(0x7fa3b00cc1f0 msgr2=0x7fa3b00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 -- 172.21.15.92:0/701114807 shutdown_connections 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.981+0000 7fa3b5a00640 1 -- 172.21.15.92:0/701114807 wait complete. 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.982+0000 7fa3b5a00640 1 Processor -- start 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.982+0000 7fa3b5a00640 1 -- start start 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3b00c5480 0x7fa3b0146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3b00c5860 0x7fa3b0146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0158ed0 con 0x7fa3b00c5860 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0071e60 con 0x7fa3b00c36f0 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.983+0000 7fa3b5a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b01591b0 con 0x7fa3b00c5480 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3af400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3b00c5480 0x7fa3b0146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51004/0 (socket says 172.21.15.92:51004) 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 -- 172.21.15.92:0/226154959 learned_addr learned my addr 172.21.15.92:0/226154959 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3b5000640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3b00c5860 0x7fa3b0146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 -- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3b00c5480 msgr2=0x7fa3b0146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3b00c5480 0x7fa3b0146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 -- 172.21.15.92:0/226154959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3b00c5860 msgr2=0x7fa3b0146580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3b00c5860 0x7fa3b0146580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 -- 172.21.15.92:0/226154959 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b014a910 con 0x7fa3b00c36f0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.984+0000 7fa3afe00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002b880 tx=0x7fa3a00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.985+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00317a0 con 0x7fa3b00c36f0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.985+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a0031940 con 0x7fa3b00c36f0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.985+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0031d10 con 0x7fa3b00c36f0 2024-03-19T02:59:09.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.985+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b014abf0 con 0x7fa3b00c36f0 2024-03-19T02:59:09.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.985+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b014b1b0 con 0x7fa3b00c36f0 2024-03-19T02:59:09.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.986+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b004f580 con 0x7fa3b00c36f0 2024-03-19T02:59:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.987+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3a0032430 con 0x7fa3b00c36f0 2024-03-19T02:59:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.988+0000 7fa3a7e00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 0x7fa39006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.988+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7fa3a002f030 con 0x7fa3b00c36f0 2024-03-19T02:59:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.988+0000 7fa3af400640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 0x7fa39006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.988+0000 7fa3af400640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 0x7fa39006e0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0147900 tx=0x7fa39c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:09.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:09.992+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3a004c060 con 0x7fa3b00c36f0 2024-03-19T02:59:10.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.228+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3b006d370 con 0x7fa3b00c36f0 2024-03-19T02:59:10.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.229+0000 7fa3a7e00640 1 -- 172.21.15.92:0/226154959 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7fa3a007b400 con 0x7fa3b00c36f0 2024-03-19T02:59:10.231 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:10.231 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:10.231 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 msgr2=0x7fa39006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 0x7fa39006e0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0147900 tx=0x7fa39c009190 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 msgr2=0x7fa3b014cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002b880 tx=0x7fa3a00046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 shutdown_connections 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa39006bc00 0x7fa39006e0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3b00c5860 0x7fa3b0146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3b00c5480 0x7fa3b0146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 --2- 172.21.15.92:0/226154959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3b00c36f0 0x7fa3b014cde0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:10.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 >> 172.21.15.92:0/226154959 conn(0x7fa3b00cc1f0 msgr2=0x7fa3b006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:10.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.232+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 shutdown_connections 2024-03-19T02:59:10.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:10.233+0000 7fa3b5a00640 1 -- 172.21.15.92:0/226154959 wait complete. 2024-03-19T02:59:10.252 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:12.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:12.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.467+0000 7f2d10000640 1 Processor -- start 2024-03-19T02:59:12.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.467+0000 7f2d10000640 1 -- start start 2024-03-19T02:59:12.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d10000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2d08072380 0x7f2d08071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d10000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080c60d0 0x7f2d080c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d10000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0815b740 con 0x7f2d08154dc0 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d10000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0815ba20 con 0x7f2d08072380 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d10000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2d080c8400 con 0x7f2d080c60d0 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d0d600640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2d08154dc0 0x7f2d08071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34080/0 (socket says 172.21.15.92:34080) 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d0d600640 1 -- 172.21.15.92:0/2688353614 learned_addr learned my addr 172.21.15.92:0/2688353614 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2688353614 >> v2:172.21.15.92:3300/0 conn(0x7f2d08072380 0x7f2d08071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.469+0000 7f2d0e000640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080c60d0 0x7f2d080c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.470+0000 7f2d07e00640 1 -- 172.21.15.92:0/2688353614 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1332825089 0 0) 0x7f2d0815b740 con 0x7f2d08154dc0 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.470+0000 7f2d0cc00640 1 -- 172.21.15.92:0/2688353614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080c60d0 msgr2=0x7f2d080c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.470+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080c60d0 0x7f2d080c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.470+0000 7f2d0cc00640 1 -- 172.21.15.92:0/2688353614 >> v1:172.21.15.92:6789/0 conn(0x7f2d08154dc0 legacy=0x7f2d08071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.470+0000 7f2d0cc00640 1 -- 172.21.15.92:0/2688353614 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d080c86e0 con 0x7f2d08072380 2024-03-19T02:59:12.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.471+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08071190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40097a0 tx=0x7f2cf402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b400a8495e510793 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:12.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.471+0000 7f2d07e00640 1 -- 172.21.15.92:0/2688353614 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4031780 con 0x7f2d08072380 2024-03-19T02:59:12.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.471+0000 7f2d07e00640 1 -- 172.21.15.92:0/2688353614 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d08072380 2024-03-19T02:59:12.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.471+0000 7f2d07e00640 1 -- 172.21.15.92:0/2688353614 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4031b40 con 0x7f2d08072380 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 -- 172.21.15.92:0/2688353614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 msgr2=0x7f2d08071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08071190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40097a0 tx=0x7f2cf402b820 comp rx=0 tx=0).stop 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 -- 172.21.15.92:0/2688353614 shutdown_connections 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d080c60d0 0x7f2d080c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 --2- 172.21.15.92:0/2688353614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08071190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 -- 172.21.15.92:0/2688353614 >> 172.21.15.92:0/2688353614 conn(0x7f2d080c8d90 msgr2=0x7f2d080c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 -- 172.21.15.92:0/2688353614 shutdown_connections 2024-03-19T02:59:12.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.472+0000 7f2d10000640 1 -- 172.21.15.92:0/2688353614 wait complete. 2024-03-19T02:59:12.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.474+0000 7f2d10000640 1 Processor -- start 2024-03-19T02:59:12.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.474+0000 7f2d10000640 1 -- start start 2024-03-19T02:59:12.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 0x7f2d0815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 0x7f2d0815a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2d080c7350 con 0x7f2d08072380 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2d080c71d0 con 0x7f2d080c60d0 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d10000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2d080c74d0 con 0x7f2d08154dc0 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:56268/0 (socket says 172.21.15.92:56268) 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 -- 172.21.15.92:0/2981631740 learned_addr learned my addr 172.21.15.92:0/2981631740 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0e000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 0x7f2d0815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 0x7f2d0815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 -- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 msgr2=0x7f2d0815a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 0x7f2d0815a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 -- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 msgr2=0x7f2d0815c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 0x7f2d0815c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0d600640 1 -- 172.21.15.92:0/2981631740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d08158020 con 0x7f2d08072380 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0e000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 0x7f2d0815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.475+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 0x7f2d0815c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d0d600640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c430 tx=0x7f2cfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:12.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0074e0 con 0x7f2d08072380 2024-03-19T02:59:12.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cfc0047e0 con 0x7f2d08072380 2024-03-19T02:59:12.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc004bd0 con 0x7f2d08072380 2024-03-19T02:59:12.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d081555a0 con 0x7f2d08072380 2024-03-19T02:59:12.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.476+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d08155b60 con 0x7f2d08072380 2024-03-19T02:59:12.480 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.477+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d080684c0 con 0x7f2d08072380 2024-03-19T02:59:12.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.478+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2cfc011070 con 0x7f2d08072380 2024-03-19T02:59:12.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.479+0000 7f2d05600640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 0x7f2ce806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:12.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.479+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f2cfc08a8e0 con 0x7f2d08072380 2024-03-19T02:59:12.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.479+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 0x7f2ce806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:12.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.480+0000 7f2d0cc00640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 0x7f2ce806e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4036000 tx=0x7f2cf4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:12.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.484+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2cfc059250 con 0x7f2d08072380 2024-03-19T02:59:12.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.725+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d08071230 con 0x7f2d08072380 2024-03-19T02:59:12.728 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.727+0000 7f2d05600640 1 -- 172.21.15.92:0/2981631740 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f2cfc058b50 con 0x7f2d08072380 2024-03-19T02:59:12.728 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:12.728 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:12.729 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 msgr2=0x7f2ce806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 0x7f2ce806e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4036000 tx=0x7f2cf4008030 comp rx=0 tx=0).stop 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 msgr2=0x7f2d08157700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c430 tx=0x7f2cfc00c8f0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 shutdown_connections 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2ce806bbd0 0x7f2ce806e090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d08154dc0 0x7f2d0815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d080c60d0 0x7f2d0815c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 --2- 172.21.15.92:0/2981631740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d08072380 0x7f2d08157700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:12.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.729+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 >> 172.21.15.92:0/2981631740 conn(0x7f2d080c8d90 msgr2=0x7f2d080c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:12.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.730+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 shutdown_connections 2024-03-19T02:59:12.732 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:12.730+0000 7f2d10000640 1 -- 172.21.15.92:0/2981631740 wait complete. 2024-03-19T02:59:12.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:14.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:14.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.972+0000 7f696f000640 1 Processor -- start 2024-03-19T02:59:14.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.972+0000 7f696f000640 1 -- start start 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696f000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696f000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f69681593d0 0x7f69680c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696f000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69680c9a80 con 0x7f69680c6470 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696f000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f69680c6c90 con 0x7f6968152bf0 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696f000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6968159910 con 0x7f69681593d0 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f6967e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f696d000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f69681593d0 0x7f69680c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f6967e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:56270/0 (socket says 172.21.15.92:56270) 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f6967e00640 1 -- 172.21.15.92:0/1508389205 learned_addr learned my addr 172.21.15.92:0/1508389205 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:14.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.974+0000 7f6967400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f69680c6470 0x7f69680c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:35520/0 (socket says 172.21.15.92:35520) 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f696d000640 1 -- 172.21.15.92:0/1508389205 >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 msgr2=0x7f696806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f696d000640 1 --2- 172.21.15.92:0/1508389205 >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f696d000640 1 -- 172.21.15.92:0/1508389205 >> v1:172.21.15.159:6789/0 conn(0x7f69680c6470 legacy=0x7f69680c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f696d000640 1 -- 172.21.15.92:0/1508389205 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69680c8650 con 0x7f69681593d0 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f6967e00640 1 --2- 172.21.15.92:0/1508389205 >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.975+0000 7f6966a00640 1 -- 172.21.15.92:0/1508389205 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680975170 0 0) 0x7f69680c9a80 con 0x7f69680c6470 2024-03-19T02:59:14.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.976+0000 7f696d000640 1 --2- 172.21.15.92:0/1508389205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69680c4690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f69680c5070 tx=0x7f696002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef2810e1223bcb7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.976+0000 7f6966a00640 1 -- 172.21.15.92:0/1508389205 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69600315a0 con 0x7f69681593d0 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.976+0000 7f6966a00640 1 -- 172.21.15.92:0/1508389205 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6960037070 con 0x7f69681593d0 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.976+0000 7f6966a00640 1 -- 172.21.15.92:0/1508389205 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960031960 con 0x7f69681593d0 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 -- 172.21.15.92:0/1508389205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 msgr2=0x7f69680c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 --2- 172.21.15.92:0/1508389205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69680c4690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f69680c5070 tx=0x7f696002b840 comp rx=0 tx=0).stop 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 -- 172.21.15.92:0/1508389205 shutdown_connections 2024-03-19T02:59:14.979 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 --2- 172.21.15.92:0/1508389205 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69680c4690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 --2- 172.21.15.92:0/1508389205 >> v2:172.21.15.92:3300/0 conn(0x7f6968152bf0 0x7f696806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 -- 172.21.15.92:0/1508389205 >> 172.21.15.92:0/1508389205 conn(0x7f69680c8d90 msgr2=0x7f69680c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 -- 172.21.15.92:0/1508389205 shutdown_connections 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.977+0000 7f696f000640 1 -- 172.21.15.92:0/1508389205 wait complete. 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.979+0000 7f696f000640 1 Processor -- start 2024-03-19T02:59:14.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.979+0000 7f696f000640 1 -- start start 2024-03-19T02:59:14.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6968152bf0 0x7f69681f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69681f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.982 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f696806ea60 con 0x7f6968152bf0 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f696806e4a0 con 0x7f69681593d0 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696f000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f696806e6f0 con 0x7f69680c6470 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f6967e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f6967400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6968152bf0 0x7f69681f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f6967e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46786/0 (socket says 172.21.15.92:46786) 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f6967e00640 1 -- 172.21.15.92:0/778496789 learned_addr learned my addr 172.21.15.92:0/778496789 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.980+0000 7f696d000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69681f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 -- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 msgr2=0x7f69681f08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69681f08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 -- 172.21.15.92:0/778496789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6968152bf0 msgr2=0x7f69681f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6968152bf0 0x7f69681f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 -- 172.21.15.92:0/778496789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69681583f0 con 0x7f69680c6470 2024-03-19T02:59:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.981+0000 7f6967e00640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f695800efc0 tx=0x7f695800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.982+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958014480 con 0x7f69680c6470 2024-03-19T02:59:14.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.982+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6958019070 con 0x7f69680c6470 2024-03-19T02:59:14.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.982+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958014820 con 0x7f69680c6470 2024-03-19T02:59:14.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.983+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69681f3af0 con 0x7f69680c6470 2024-03-19T02:59:14.984 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.983+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69681f40b0 con 0x7f69680c6470 2024-03-19T02:59:14.985 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.984+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69680c3b50 con 0x7f69680c6470 2024-03-19T02:59:14.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.985+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6958005d10 con 0x7f69680c6470 2024-03-19T02:59:14.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.985+0000 7f695fe00640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 0x7f694406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.985+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f695808b060 con 0x7f69680c6470 2024-03-19T02:59:14.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.986+0000 7f6967400640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 0x7f694406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.986+0000 7f6967400640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 0x7f694406e0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6968158e70 tx=0x7f6950007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:14.990+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6958010040 con 0x7f69680c6470 2024-03-19T02:59:15.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.224+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f696806f950 con 0x7f69680c6470 2024-03-19T02:59:15.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.225+0000 7f695fe00640 1 -- 172.21.15.92:0/778496789 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f69580592f0 con 0x7f69680c6470 2024-03-19T02:59:15.227 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:15.227 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:15.227 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:15.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.227+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 msgr2=0x7f694406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.227+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 0x7f694406e0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6968158e70 tx=0x7f6950007d50 comp rx=0 tx=0).stop 2024-03-19T02:59:15.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.227+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 msgr2=0x7f69680cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.227+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f695800efc0 tx=0x7f695800c490 comp rx=0 tx=0).stop 2024-03-19T02:59:15.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 shutdown_connections 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f694406bc00 0x7f694406e0c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69681593d0 0x7f69681f08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6968152bf0 0x7f69681f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 --2- 172.21.15.92:0/778496789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69680c6470 0x7f69680cba40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 >> 172.21.15.92:0/778496789 conn(0x7f69680c8d90 msgr2=0x7f69680c4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 shutdown_connections 2024-03-19T02:59:15.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:15.228+0000 7f696f000640 1 -- 172.21.15.92:0/778496789 wait complete. 2024-03-19T02:59:15.246 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:17.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:17.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.464+0000 7f17cf200640 1 Processor -- start 2024-03-19T02:59:17.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.464+0000 7f17cf200640 1 -- start start 2024-03-19T02:59:17.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17cf200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f17c806f800 0x7f17c80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17cf200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c8072890 con 0x7f17c8071da0 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17cf200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c80c8200 con 0x7f17c8158db0 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17cf200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f17c80c6c90 con 0x7f17c806f800 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f17c8071da0 0x7f17c8072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33686/0 (socket says 172.21.15.92:33686) 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7e00640 1 -- 172.21.15.92:0/40968415 learned_addr learned my addr 172.21.15.92:0/40968415 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17bf400640 1 --2- 172.21.15.92:0/40968415 >> v2:172.21.15.159:3300/0 conn(0x7f17c806f800 0x7f17c80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699911706 0 0) 0x7f17c8072890 con 0x7f17c8071da0 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17ac003520 con 0x7f17c8071da0 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613366645 0 0) 0x7f17c80c8200 con 0x7f17c8158db0 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c8072890 con 0x7f17c8158db0 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17bf400640 1 -- 172.21.15.92:0/40968415 >> v1:172.21.15.159:6789/0 conn(0x7f17c8158db0 legacy=0x7f17c80c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17bf400640 1 -- 172.21.15.92:0/40968415 >> v1:172.21.15.92:6789/0 conn(0x7f17c8071da0 legacy=0x7f17c8072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.467+0000 7f17bf400640 1 -- 172.21.15.92:0/40968415 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c80c84e0 con 0x7f17c806f800 2024-03-19T02:59:17.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.468+0000 7f17bf400640 1 --2- 172.21.15.92:0/40968415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c80c8740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f17b00097a0 tx=0x7f17b002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c66623319ebd49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:17.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.468+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0034070 con 0x7f17c806f800 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.468+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b0030610 con 0x7f17c806f800 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.468+0000 7f17c7400640 1 -- 172.21.15.92:0/40968415 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b00309c0 con 0x7f17c806f800 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 -- 172.21.15.92:0/40968415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 msgr2=0x7f17c80c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 --2- 172.21.15.92:0/40968415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c80c8740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f17b00097a0 tx=0x7f17b002b960 comp rx=0 tx=0).stop 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 -- 172.21.15.92:0/40968415 shutdown_connections 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 --2- 172.21.15.92:0/40968415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c80c8740 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 -- 172.21.15.92:0/40968415 >> 172.21.15.92:0/40968415 conn(0x7f17c80c8d90 msgr2=0x7f17c80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 -- 172.21.15.92:0/40968415 shutdown_connections 2024-03-19T02:59:17.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.469+0000 7f17cf200640 1 -- 172.21.15.92:0/40968415 wait complete. 2024-03-19T02:59:17.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.471+0000 7f17cf200640 1 Processor -- start 2024-03-19T02:59:17.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.471+0000 7f17cf200640 1 -- start start 2024-03-19T02:59:17.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c8160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:17.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17c8158db0 0x7f17c815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f17c80c7830 con 0x7f17c8071da0 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f17c815fde0 con 0x7f17c806f800 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cf200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f17c80c79b0 con 0x7f17c8158db0 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17bf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17bf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45996/0 (socket says 172.21.15.92:45996) 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17c7e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c8160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17bf400640 1 -- 172.21.15.92:0/1942794512 learned_addr learned my addr 172.21.15.92:0/1942794512 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:17.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.472+0000 7f17cd200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17c8158db0 0x7f17c815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:17.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 -- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17c8158db0 msgr2=0x7f17c815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17c8158db0 0x7f17c815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 -- 172.21.15.92:0/1942794512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 msgr2=0x7f17c8154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 -- 172.21.15.92:0/1942794512 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c815ebf0 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17bf400640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c7e00640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c8160180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c3e0 tx=0x7f17b800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8017070 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b8005ce0 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80072d0 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c815eeb0 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.473+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c81619f0 con 0x7f17c806f800 2024-03-19T02:59:17.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.474+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c80c6980 con 0x7f17c806f800 2024-03-19T02:59:17.477 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.475+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f17b801a070 con 0x7f17c806f800 2024-03-19T02:59:17.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.476+0000 7f17c4c00640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 0x7f17a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:17.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.476+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f17b8089e50 con 0x7f17c806f800 2024-03-19T02:59:17.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.477+0000 7f17bf400640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 0x7f17a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:17.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.477+0000 7f17bf400640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 0x7f17a406e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17c806e290 tx=0x7f17b003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:17.482 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.482+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f17c80c6980 con 0x7f17c806f800 2024-03-19T02:59:17.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.711+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17c806f360 con 0x7f17c806f800 2024-03-19T02:59:17.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.712+0000 7f17c4c00640 1 -- 172.21.15.92:0/1942794512 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f17b8058200 con 0x7f17c806f800 2024-03-19T02:59:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:17.712 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:17.713 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 msgr2=0x7f17a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 0x7f17a406e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17c806e290 tx=0x7f17b003b040 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 msgr2=0x7f17c8160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c8160180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c3e0 tx=0x7f17b800c8a0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 shutdown_connections 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f17a406bbd0 0x7f17a406e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f17c8158db0 0x7f17c815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17c8071da0 0x7f17c8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 --2- 172.21.15.92:0/1942794512 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f17c806f800 0x7f17c8160180 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.714+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 >> 172.21.15.92:0/1942794512 conn(0x7f17c80c8d90 msgr2=0x7f17c81568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:17.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.715+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 shutdown_connections 2024-03-19T02:59:17.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:17.715+0000 7f17cf200640 1 -- 172.21.15.92:0/1942794512 wait complete. 2024-03-19T02:59:17.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:19.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:19.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.956+0000 7f59e4200640 1 Processor -- start 2024-03-19T02:59:19.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.956+0000 7f59e4200640 1 -- start start 2024-03-19T02:59:19.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e4200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 0x7f59dc071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:19.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e4200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 0x7f59dc0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:19.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e4200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59dc15b740 con 0x7f59dc0c60d0 2024-03-19T02:59:19.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e4200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f59dc15ba20 con 0x7f59dc072380 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e4200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc0c8400 con 0x7f59dc154dc0 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e0e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f59dc0c60d0 0x7f59dc071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33690/0 (socket says 172.21.15.92:33690) 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e0e00640 1 -- 172.21.15.92:0/517935887 learned_addr learned my addr 172.21.15.92:0/517935887 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e2200640 1 --2- 172.21.15.92:0/517935887 >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 0x7f59dc0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.958+0000 7f59e1800640 1 --2- 172.21.15.92:0/517935887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 0x7f59dc071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59e1800640 1 -- 172.21.15.92:0/517935887 >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 msgr2=0x7f59dc0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59e1800640 1 --2- 172.21.15.92:0/517935887 >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 0x7f59dc0c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:19.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59dbe00640 1 -- 172.21.15.92:0/517935887 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120746023 0 0) 0x7f59dc15b740 con 0x7f59dc0c60d0 2024-03-19T02:59:19.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59e1800640 1 -- 172.21.15.92:0/517935887 >> v1:172.21.15.92:6789/0 conn(0x7f59dc0c60d0 legacy=0x7f59dc071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:19.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59e1800640 1 -- 172.21.15.92:0/517935887 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc0c86e0 con 0x7f59dc154dc0 2024-03-19T02:59:19.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.959+0000 7f59e2200640 1 --2- 172.21.15.92:0/517935887 >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 0x7f59dc0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59e1800640 1 --2- 172.21.15.92:0/517935887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 0x7f59dc071190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59d00097a0 tx=0x7f59d002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42288036d9523db6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59dbe00640 1 -- 172.21.15.92:0/517935887 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0031780 con 0x7f59dc154dc0 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59dbe00640 1 -- 172.21.15.92:0/517935887 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d0037070 con 0x7f59dc154dc0 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59dbe00640 1 -- 172.21.15.92:0/517935887 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0031b40 con 0x7f59dc154dc0 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59e4200640 1 -- 172.21.15.92:0/517935887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 msgr2=0x7f59dc071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.960+0000 7f59e4200640 1 --2- 172.21.15.92:0/517935887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 0x7f59dc071190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59d00097a0 tx=0x7f59d002b820 comp rx=0 tx=0).stop 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 -- 172.21.15.92:0/517935887 shutdown_connections 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 --2- 172.21.15.92:0/517935887 >> v2:172.21.15.159:3300/0 conn(0x7f59dc072380 0x7f59dc0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:19.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 --2- 172.21.15.92:0/517935887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc154dc0 0x7f59dc071190 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:19.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 -- 172.21.15.92:0/517935887 >> 172.21.15.92:0/517935887 conn(0x7f59dc0c8d90 msgr2=0x7f59dc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:19.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 -- 172.21.15.92:0/517935887 shutdown_connections 2024-03-19T02:59:19.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.961+0000 7f59e4200640 1 -- 172.21.15.92:0/517935887 wait complete. 2024-03-19T02:59:19.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.962+0000 7f59e4200640 1 Processor -- start 2024-03-19T02:59:19.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.962+0000 7f59e4200640 1 -- start start 2024-03-19T02:59:19.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:19.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59dc0c60d0 0x7f59dc1f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59dc154dc0 0x7f59dc1f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0c7350 con 0x7f59dc0c60d0 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0c71d0 con 0x7f59dc154dc0 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.963+0000 7f59e4200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc0c74d0 con 0x7f59dc072380 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:50970/0 (socket says 172.21.15.92:50970) 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 -- 172.21.15.92:0/4187885458 learned_addr learned my addr 172.21.15.92:0/4187885458 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e0e00640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59dc0c60d0 0x7f59dc1f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e2200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59dc154dc0 0x7f59dc1f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 -- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59dc154dc0 msgr2=0x7f59dc1f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59dc154dc0 0x7f59dc1f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 -- 172.21.15.92:0/4187885458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59dc0c60d0 msgr2=0x7f59dc1f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59dc0c60d0 0x7f59dc1f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 -- 172.21.15.92:0/4187885458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc15ee00 con 0x7f59dc072380 2024-03-19T02:59:19.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.964+0000 7f59e1800640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f59d0034040 tx=0x7f59d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:19.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.965+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0004030 con 0x7f59dc072380 2024-03-19T02:59:19.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.965+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00041d0 con 0x7f59dc072380 2024-03-19T02:59:19.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.965+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0003610 con 0x7f59dc072380 2024-03-19T02:59:19.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.965+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc1f0360 con 0x7f59dc072380 2024-03-19T02:59:19.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.965+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc15e880 con 0x7f59dc072380 2024-03-19T02:59:20.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.966+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc0684c0 con 0x7f59dc072380 2024-03-19T02:59:20.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.967+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f59d00037b0 con 0x7f59dc072380 2024-03-19T02:59:20.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.968+0000 7f59d9600640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:20.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.968+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f59d00acd00 con 0x7f59dc072380 2024-03-19T02:59:20.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.968+0000 7f59e0e00640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:20.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.969+0000 7f59e0e00640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f59dc14fc70 tx=0x7f59c8007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:20.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:19.971+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59dc1f0360 con 0x7f59dc072380 2024-03-19T02:59:20.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.199+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59dc15eb20 con 0x7f59dc072380 2024-03-19T02:59:20.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.200+0000 7f59d9600640 1 -- 172.21.15.92:0/4187885458 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f59d0041070 con 0x7f59dc072380 2024-03-19T02:59:20.201 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:20.201 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:20.201 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 msgr2=0x7f59bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f59dc14fc70 tx=0x7f59c8007d50 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 msgr2=0x7f59dc1f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f59d0034040 tx=0x7f59d00046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 shutdown_connections 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59dc154dc0 0x7f59dc1f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59dc0c60d0 0x7f59dc1f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 --2- 172.21.15.92:0/4187885458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59dc072380 0x7f59dc1f4290 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:20.204 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 >> 172.21.15.92:0/4187885458 conn(0x7f59dc0c8d90 msgr2=0x7f59dc0c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:20.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 shutdown_connections 2024-03-19T02:59:20.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:20.203+0000 7f59e4200640 1 -- 172.21.15.92:0/4187885458 wait complete. 2024-03-19T02:59:20.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:22.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:22.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.441+0000 7f0546800640 1 Processor -- start 2024-03-19T02:59:22.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.441+0000 7f0546800640 1 -- start start 2024-03-19T02:59:22.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.442+0000 7f0546800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05401593d0 0x7f05400c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:22.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.442+0000 7f0546800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05400c9a80 con 0x7f0540152bf0 2024-03-19T02:59:22.443 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.442+0000 7f0546800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05400c8650 con 0x7f05400c6470 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.442+0000 7f0546800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f05400c6c90 con 0x7f05401593d0 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05401593d0 0x7f05400c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0540152bf0 0x7f054006eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33702/0 (socket says 172.21.15.92:33702) 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053ea00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f05400c6470 0x7f05400c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55328/0 (socket says 172.21.15.92:55328) 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053f400640 1 -- 172.21.15.92:0/2482380072 learned_addr learned my addr 172.21.15.92:0/2482380072 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053e000640 1 -- 172.21.15.92:0/2482380072 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364862046 0 0) 0x7f05400c8650 con 0x7f05400c6470 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053fe00640 1 -- 172.21.15.92:0/2482380072 >> v1:172.21.15.159:6789/0 conn(0x7f05400c6470 legacy=0x7f05400c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053fe00640 1 -- 172.21.15.92:0/2482380072 >> v1:172.21.15.92:6789/0 conn(0x7f0540152bf0 legacy=0x7f054006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.443+0000 7f053fe00640 1 -- 172.21.15.92:0/2482380072 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05400c8930 con 0x7f05401593d0 2024-03-19T02:59:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f053fe00640 1 --2- 172.21.15.92:0/2482380072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f05400c4690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0538009d90 tx=0x7f053802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731783de4d48b81e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f053e000640 1 -- 172.21.15.92:0/2482380072 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538039070 con 0x7f05401593d0 2024-03-19T02:59:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f053e000640 1 -- 172.21.15.92:0/2482380072 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053802cc60 con 0x7f05401593d0 2024-03-19T02:59:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f053e000640 1 -- 172.21.15.92:0/2482380072 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538034660 con 0x7f05401593d0 2024-03-19T02:59:22.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f0546800640 1 -- 172.21.15.92:0/2482380072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 msgr2=0x7f05400c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.445+0000 7f0546800640 1 --2- 172.21.15.92:0/2482380072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f05400c4690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0538009d90 tx=0x7f053802beb0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.446+0000 7f0546800640 1 -- 172.21.15.92:0/2482380072 shutdown_connections 2024-03-19T02:59:22.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.446+0000 7f0546800640 1 --2- 172.21.15.92:0/2482380072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f05400c4690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.446+0000 7f0546800640 1 -- 172.21.15.92:0/2482380072 >> 172.21.15.92:0/2482380072 conn(0x7f05400c8d90 msgr2=0x7f05400c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:22.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.446+0000 7f0546800640 1 -- 172.21.15.92:0/2482380072 shutdown_connections 2024-03-19T02:59:22.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.446+0000 7f0546800640 1 -- 172.21.15.92:0/2482380072 wait complete. 2024-03-19T02:59:22.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.447+0000 7f0546800640 1 Processor -- start 2024-03-19T02:59:22.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.447+0000 7f0546800640 1 -- start start 2024-03-19T02:59:22.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05400c6470 0x7f0540156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:22.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0540152bf0 0x7f0540072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:22.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:22.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f054006e4a0 con 0x7f05401593d0 2024-03-19T02:59:22.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0540159910 con 0x7f0540152bf0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.448+0000 7f0546800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f054006ea60 con 0x7f05400c6470 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05400c6470 0x7f0540156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0540152bf0 0x7f0540072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:46012/0 (socket says 172.21.15.92:46012) 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 -- 172.21.15.92:0/1801286421 learned_addr learned my addr 172.21.15.92:0/1801286421 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 -- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05400c6470 msgr2=0x7f0540156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05400c6470 0x7f0540156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 -- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0540152bf0 msgr2=0x7f0540072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0540152bf0 0x7f0540072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 -- 172.21.15.92:0/1801286421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054014ea20 con 0x7f05401593d0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.449+0000 7f053fe00640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0538037040 tx=0x7f0538004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:22.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.450+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538039050 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.450+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053803a040 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.450+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054014ecc0 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.450+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053803e610 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.450+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0540153770 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.451+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05400684c0 con 0x7f05401593d0 2024-03-19T02:59:22.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.452+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0538047020 con 0x7f05401593d0 2024-03-19T02:59:22.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.453+0000 7f0537400640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 0x7f052006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:22.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.453+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5968+0+0 (secure 0 0 0) 0x7f05380ace60 con 0x7f05401593d0 2024-03-19T02:59:22.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.453+0000 7f053f400640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 0x7f052006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:22.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.454+0000 7f053f400640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 0x7f052006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0530005880 tx=0x7f0530009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:22.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.457+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f053804e020 con 0x7f05401593d0 2024-03-19T02:59:22.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.689+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05400c3b50 con 0x7f05401593d0 2024-03-19T02:59:22.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.690+0000 7f0537400640 1 -- 172.21.15.92:0/1801286421 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6838 (secure 0 0 0) 0x7f053807b240 con 0x7f05401593d0 2024-03-19T02:59:22.691 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:22.691 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4688,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:57:30.109752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"f","rank":0,"incarnation":9,"state":"up:active","state_seq":42,"addr":"172.21.15.159:6837/2387435110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2387435110},{"type":"v1","addr":"172.21.15.159:6837","nonce":2387435110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-03-19T02:59:22.691 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 14 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 msgr2=0x7f052006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 0x7f052006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0530005880 tx=0x7f0530009040 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 msgr2=0x7f0540146b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0538037040 tx=0x7f0538004a40 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 shutdown_connections 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f052006bbd0 0x7f052006e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05401593d0 0x7f0540146b70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0540152bf0 0x7f0540072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 --2- 172.21.15.92:0/1801286421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f05400c6470 0x7f0540156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 >> 172.21.15.92:0/1801286421 conn(0x7f05400c8d90 msgr2=0x7f05401501a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 shutdown_connections 2024-03-19T02:59:22.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:22.693+0000 7f0546800640 1 -- 172.21.15.92:0/1801286421 wait complete. 2024-03-19T02:59:22.710 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T02:59:24.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:24.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.924+0000 7fcb96600640 1 Processor -- start 2024-03-19T02:59:24.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.924+0000 7fcb96600640 1 -- start start 2024-03-19T02:59:24.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb96600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fcb90071da0 0x7fcb900c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb96600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb90158db0 0x7fcb90159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb96600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb90072890 con 0x7fcb9006f800 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb96600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcb900c6c90 con 0x7fcb90158db0 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb96600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fcb900c8200 con 0x7fcb90071da0 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb8fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb90158db0 0x7fcb90159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb8ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fcb9006f800 0x7fcb90072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33712/0 (socket says 172.21.15.92:33712) 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb8ea00640 1 -- 172.21.15.92:0/2695499413 learned_addr learned my addr 172.21.15.92:0/2695499413 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb8fe00640 1 --2- 172.21.15.92:0/2695499413 >> v2:172.21.15.92:3300/0 conn(0x7fcb90158db0 0x7fcb90159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:46022/0 (socket says 172.21.15.92:46022) 2024-03-19T02:59:24.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.926+0000 7fcb8f400640 1 --2- 172.21.15.92:0/2695499413 >> v2:172.21.15.159:3300/0 conn(0x7fcb90071da0 0x7fcb900c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.927+0000 7fcb8fe00640 1 -- 172.21.15.92:0/2695499413 >> v2:172.21.15.159:3300/0 conn(0x7fcb90071da0 msgr2=0x7fcb900c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.927+0000 7fcb8fe00640 1 --2- 172.21.15.92:0/2695499413 >> v2:172.21.15.159:3300/0 conn(0x7fcb90071da0 0x7fcb900c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.927+0000 7fcb8fe00640 1 -- 172.21.15.92:0/2695499413 >> v1:172.21.15.92:6789/0 conn(0x7fcb9006f800 legacy=0x7fcb90072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.927+0000 7fcb8e000640 1 -- 172.21.15.92:0/2695499413 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215170651 0 0) 0x7fcb90072890 con 0x7fcb9006f800 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.927+0000 7fcb8fe00640 1 -- 172.21.15.92:0/2695499413 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb90159740 con 0x7fcb90158db0 2024-03-19T02:59:24.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.928+0000 7fcb8fe00640 1 --2- 172.21.15.92:0/2695499413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90158db0 0x7fcb90159200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb90075650 tx=0x7fcb8802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cbe072ccccc19 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:24.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.928+0000 7fcb8e000640 1 -- 172.21.15.92:0/2695499413 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88039070 con 0x7fcb90158db0 2024-03-19T02:59:24.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.928+0000 7fcb8e000640 1 -- 172.21.15.92:0/2695499413 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb8802cc80 con 0x7fcb90158db0 2024-03-19T02:59:24.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.928+0000 7fcb8e000640 1 -- 172.21.15.92:0/2695499413 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88034660 con 0x7fcb90158db0 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 -- 172.21.15.92:0/2695499413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90158db0 msgr2=0x7fcb90159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 --2- 172.21.15.92:0/2695499413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90158db0 0x7fcb90159200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb90075650 tx=0x7fcb8802bed0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 -- 172.21.15.92:0/2695499413 shutdown_connections 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 --2- 172.21.15.92:0/2695499413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90158db0 0x7fcb90159200 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 --2- 172.21.15.92:0/2695499413 >> v2:172.21.15.159:3300/0 conn(0x7fcb90071da0 0x7fcb900c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 -- 172.21.15.92:0/2695499413 >> 172.21.15.92:0/2695499413 conn(0x7fcb900c8d90 msgr2=0x7fcb900c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 -- 172.21.15.92:0/2695499413 shutdown_connections 2024-03-19T02:59:24.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.929+0000 7fcb96600640 1 -- 172.21.15.92:0/2695499413 wait complete. 2024-03-19T02:59:24.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.930+0000 7fcb96600640 1 Processor -- start 2024-03-19T02:59:24.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.931+0000 7fcb96600640 1 -- start start 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.931+0000 7fcb96600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb9006f800 0x7fcb900cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb96600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb96600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb90158db0 0x7fcb901f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb96600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9015fde0 con 0x7fcb90071da0 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb96600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9015fb00 con 0x7fcb90158db0 2024-03-19T02:59:24.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb96600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcb900c7120 con 0x7fcb9006f800 2024-03-19T02:59:24.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:46038/0 (socket says 172.21.15.92:46038) 2024-03-19T02:59:24.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 -- 172.21.15.92:0/132382320 learned_addr learned my addr 172.21.15.92:0/132382320 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:24.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb9006f800 0x7fcb900cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb90158db0 0x7fcb901f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 -- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb9006f800 msgr2=0x7fcb900cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:24.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb9006f800 0x7fcb900cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 -- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb90158db0 msgr2=0x7fcb901f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:24.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb90158db0 0x7fcb901f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:24.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.932+0000 7fcb8ea00640 1 -- 172.21.15.92:0/132382320 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb90149640 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb8ea00640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c500 tx=0x7fcb7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c018070 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb7c007580 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9014e100 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c004a20 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.933+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb90149080 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.934+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb90068de0 con 0x7fcb90071da0 2024-03-19T02:59:24.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.935+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcb7c01b070 con 0x7fcb90071da0 2024-03-19T02:59:24.937 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.936+0000 7fcb87400640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 0x7fcb7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:24.937 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.936+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7fcb7c089e10 con 0x7fcb90071da0 2024-03-19T02:59:24.937 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.936+0000 7fcb8f400640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 0x7fcb7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:24.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.937+0000 7fcb8f400640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 0x7fcb7006e0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009b40 tx=0x7fcb80009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:24.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:24.940+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcb90068de0 con 0x7fcb90071da0 2024-03-19T02:59:25.178 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.177+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb900684c0 con 0x7fcb90071da0 2024-03-19T02:59:25.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.178+0000 7fcb87400640 1 -- 172.21.15.92:0/132382320 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6003 (secure 0 0 0) 0x7fcb7c08d030 con 0x7fcb90071da0 2024-03-19T02:59:25.179 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:25.179 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:24.630720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:25.180 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 15 2024-03-19T02:59:25.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.180+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 msgr2=0x7fcb7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.180+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 0x7fcb7006e0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009b40 tx=0x7fcb80009190 comp rx=0 tx=0).stop 2024-03-19T02:59:25.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.180+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 msgr2=0x7fcb90155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.180+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c500 tx=0x7fcb7c00c9c0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 shutdown_connections 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fcb7006bc00 0x7fcb7006e0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb90158db0 0x7fcb901f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb90071da0 0x7fcb90155df0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 --2- 172.21.15.92:0/132382320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb9006f800 0x7fcb900cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 >> 172.21.15.92:0/132382320 conn(0x7fcb900c8d90 msgr2=0x7fcb900c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 shutdown_connections 2024-03-19T02:59:25.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.181+0000 7fcb96600640 1 -- 172.21.15.92:0/132382320 wait complete. 2024-03-19T02:59:25.201 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-19T02:59:25.201 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-19T02:59:25.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:25.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.409+0000 7f3255000640 1 Processor -- start 2024-03-19T02:59:25.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.409+0000 7f3255000640 1 -- start start 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.410+0000 7f3255000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3250151900 0x7f3250158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.410+0000 7f3255000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f32500c7da0 0x7f32500c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.410+0000 7f3255000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32500c3b80 con 0x7f3250151ce0 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.410+0000 7f3255000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3250158860 con 0x7f32500c7da0 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.410+0000 7f3255000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f3250072e70 con 0x7f3250151900 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324de00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3250151900 0x7f3250158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324d400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3250151ce0 0x7f32500723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55338/0 (socket says 172.21.15.92:55338) 2024-03-19T02:59:25.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324d400640 1 -- 172.21.15.92:0/1385879349 learned_addr learned my addr 172.21.15.92:0/1385879349 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:25.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324e800640 1 --2- 172.21.15.92:0/1385879349 >> v2:172.21.15.92:3300/0 conn(0x7f32500c7da0 0x7f32500c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324de00640 1 -- 172.21.15.92:0/1385879349 >> v2:172.21.15.92:3300/0 conn(0x7f32500c7da0 msgr2=0x7f32500c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:25.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324de00640 1 --2- 172.21.15.92:0/1385879349 >> v2:172.21.15.92:3300/0 conn(0x7f32500c7da0 0x7f32500c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324de00640 1 -- 172.21.15.92:0/1385879349 >> v1:172.21.15.159:6789/0 conn(0x7f3250151ce0 legacy=0x7f32500723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324de00640 1 -- 172.21.15.92:0/1385879349 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3250073150 con 0x7f3250151900 2024-03-19T02:59:25.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.411+0000 7f324ca00640 1 -- 172.21.15.92:0/1385879349 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43623458 0 0) 0x7f32500c3b80 con 0x7f3250151ce0 2024-03-19T02:59:25.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.412+0000 7f324de00640 1 --2- 172.21.15.92:0/1385879349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151900 0x7f3250158320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3250075650 tx=0x7f324002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47de59ab8e2b46b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f324ca00640 1 -- 172.21.15.92:0/1385879349 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240034070 con 0x7f3250151900 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f324ca00640 1 -- 172.21.15.92:0/1385879349 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3240030610 con 0x7f3250151900 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f324ca00640 1 -- 172.21.15.92:0/1385879349 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32400309e0 con 0x7f3250151900 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 -- 172.21.15.92:0/1385879349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151900 msgr2=0x7f3250158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 --2- 172.21.15.92:0/1385879349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151900 0x7f3250158320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3250075650 tx=0x7f324002b960 comp rx=0 tx=0).stop 2024-03-19T02:59:25.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 -- 172.21.15.92:0/1385879349 shutdown_connections 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 --2- 172.21.15.92:0/1385879349 >> v2:172.21.15.92:3300/0 conn(0x7f32500c7da0 0x7f32500c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 --2- 172.21.15.92:0/1385879349 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151900 0x7f3250158320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 -- 172.21.15.92:0/1385879349 >> 172.21.15.92:0/1385879349 conn(0x7f32500c8d90 msgr2=0x7f32500c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.413+0000 7f3255000640 1 -- 172.21.15.92:0/1385879349 shutdown_connections 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.414+0000 7f3255000640 1 -- 172.21.15.92:0/1385879349 wait complete. 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.415+0000 7f3255000640 1 Processor -- start 2024-03-19T02:59:25.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.415+0000 7f3255000640 1 -- start start 2024-03-19T02:59:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3250151900 0x7f32501450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151ce0 0x7f32501455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f32500c9af0 con 0x7f3250151900 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3250071eb0 con 0x7f3250151ce0 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.416+0000 7f3255000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f32500c9c70 con 0x7f32500c7da0 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:51008/0 (socket says 172.21.15.92:51008) 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 -- 172.21.15.92:0/2044733059 learned_addr learned my addr 172.21.15.92:0/2044733059 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324e800640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151ce0 0x7f32501455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324d400640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3250151900 0x7f32501450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 -- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151ce0 msgr2=0x7f32501455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151ce0 0x7f32501455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 -- 172.21.15.92:0/2044733059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3250151900 msgr2=0x7f32501450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3250151900 0x7f32501450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 -- 172.21.15.92:0/2044733059 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325014fda0 con 0x7f32500c7da0 2024-03-19T02:59:25.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.417+0000 7f324de00640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3240000c00 tx=0x7f32400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:25.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.418+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240034040 con 0x7f32500c7da0 2024-03-19T02:59:25.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.418+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3240030410 con 0x7f32500c7da0 2024-03-19T02:59:25.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.418+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240039c50 con 0x7f32500c7da0 2024-03-19T02:59:25.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.418+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3250152460 con 0x7f32500c7da0 2024-03-19T02:59:25.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.418+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3250153640 con 0x7f32500c7da0 2024-03-19T02:59:25.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.419+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3250150460 con 0x7f32500c7da0 2024-03-19T02:59:25.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.420+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3240037070 con 0x7f32500c7da0 2024-03-19T02:59:25.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.421+0000 7f3246000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 0x7f322c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:25.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.421+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f32400acae0 con 0x7f32500c7da0 2024-03-19T02:59:25.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.421+0000 7f324d400640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 0x7f322c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:25.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.422+0000 7f324d400640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 0x7f322c06e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32501f2ad0 tx=0x7f3238007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:25.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.426+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3250150460 con 0x7f32500c7da0 2024-03-19T02:59:25.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.669+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32500c5340 con 0x7f32500c7da0 2024-03-19T02:59:25.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.669+0000 7f3246000640 1 -- 172.21.15.92:0/2044733059 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6003 (secure 0 0 0) 0x7f32500c5340 con 0x7f32500c7da0 2024-03-19T02:59:25.671 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:25.671 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:24.630720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:25.672 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 15 2024-03-19T02:59:25.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.672+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 msgr2=0x7f322c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.672+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 0x7f322c06e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f32501f2ad0 tx=0x7f3238007d50 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 msgr2=0x7f325014cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3240000c00 tx=0x7f32400046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 shutdown_connections 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322c06bbd0 0x7f322c06e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3250151ce0 0x7f32501455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3250151900 0x7f32501450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 --2- 172.21.15.92:0/2044733059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32500c7da0 0x7f325014cee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 >> 172.21.15.92:0/2044733059 conn(0x7f32500c8d90 msgr2=0x7f3250072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 shutdown_connections 2024-03-19T02:59:25.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:25.673+0000 7f3255000640 1 -- 172.21.15.92:0/2044733059 wait complete. 2024-03-19T02:59:25.692 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4688": { "addr": "172.21.15.159:6839/2444776025", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6838", "nonce": 2444776025, "type": "v2" }, { "addr": "172.21.15.159:6839", "nonce": 2444776025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4688, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T02:59:24.630720+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4688 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T02:59:27.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:27.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.905+0000 7f45b1200640 1 Processor -- start 2024-03-19T02:59:27.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.905+0000 7f45b1200640 1 -- start start 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.906+0000 7f45b1200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f45ac0c6470 0x7f45ac0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45b1200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac0c9a80 con 0x7f45ac152bf0 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45b1200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac0c8650 con 0x7f45ac1593d0 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45b1200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac0c6c90 con 0x7f45ac0c6470 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f45ac152bf0 0x7f45ac06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:43586/0 (socket says 172.21.15.92:43586) 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9e00640 1 -- 172.21.15.92:0/2231425369 learned_addr learned my addr 172.21.15.92:0/2231425369 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a3e00640 1 --2- 172.21.15.92:0/2231425369 >> v2:172.21.15.159:3300/0 conn(0x7f45ac0c6470 0x7f45ac0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474756661 0 0) 0x7f45ac0c9a80 con 0x7f45ac152bf0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4590003520 con 0x7f45ac152bf0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778810202 0 0) 0x7f45ac0c8650 con 0x7f45ac1593d0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.907+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45ac0c9a80 con 0x7f45ac1593d0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.908+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3287229810 0 0) 0x7f4590003520 con 0x7f45ac152bf0 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.908+0000 7f45a3e00640 1 -- 172.21.15.92:0/2231425369 >> v1:172.21.15.159:6789/0 conn(0x7f45ac1593d0 legacy=0x7f45ac0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.908+0000 7f45a3e00640 1 -- 172.21.15.92:0/2231425369 >> v1:172.21.15.92:6789/0 conn(0x7f45ac152bf0 legacy=0x7f45ac06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:27.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.908+0000 7f45a3e00640 1 -- 172.21.15.92:0/2231425369 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac0c8930 con 0x7f45ac0c6470 2024-03-19T02:59:27.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.909+0000 7f45a3e00640 1 --2- 172.21.15.92:0/2231425369 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac0c3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f459402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41250d504a87128b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:27.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.909+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594037070 con 0x7f45ac0c6470 2024-03-19T02:59:27.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.909+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4594030510 con 0x7f45ac0c6470 2024-03-19T02:59:27.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45a9400640 1 -- 172.21.15.92:0/2231425369 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c001670 con 0x7f45ac0c6470 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 -- 172.21.15.92:0/2231425369 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 msgr2=0x7f45ac0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 --2- 172.21.15.92:0/2231425369 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac0c3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f459402b790 comp rx=0 tx=0).stop 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 -- 172.21.15.92:0/2231425369 shutdown_connections 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 --2- 172.21.15.92:0/2231425369 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac0c3f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 -- 172.21.15.92:0/2231425369 >> 172.21.15.92:0/2231425369 conn(0x7f45ac0c8d90 msgr2=0x7f45ac0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 -- 172.21.15.92:0/2231425369 shutdown_connections 2024-03-19T02:59:27.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.910+0000 7f45b1200640 1 -- 172.21.15.92:0/2231425369 wait complete. 2024-03-19T02:59:27.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.912+0000 7f45b1200640 1 Processor -- start 2024-03-19T02:59:27.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.912+0000 7f45b1200640 1 -- start start 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ac1593d0 0x7f45ac071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac06e4a0 con 0x7f45ac152bf0 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac159910 con 0x7f45ac0c6470 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45b1200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac06ea60 con 0x7f45ac1593d0 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:35558/0 (socket says 172.21.15.92:35558) 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45a3e00640 1 -- 172.21.15.92:0/1209755021 learned_addr learned my addr 172.21.15.92:0/1209755021 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45aa800640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ac1593d0 0x7f45ac071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:27.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.913+0000 7f45a9e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 -- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ac1593d0 msgr2=0x7f45ac071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ac1593d0 0x7f45ac071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 -- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 msgr2=0x7f45ac142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 -- 172.21.15.92:0/1209755021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac1f50a0 con 0x7f45ac152bf0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a3e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f459402b7d0 tx=0x7f45940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.914+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594042020 con 0x7f45ac152bf0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.915+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45940044c0 con 0x7f45ac152bf0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.915+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594037040 con 0x7f45ac152bf0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.915+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac071e40 con 0x7f45ac152bf0 2024-03-19T02:59:27.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.915+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac145520 con 0x7f45ac152bf0 2024-03-19T02:59:27.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.916+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ac157440 con 0x7f45ac152bf0 2024-03-19T02:59:27.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.917+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4594045020 con 0x7f45ac152bf0 2024-03-19T02:59:27.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.918+0000 7f45a2000640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 0x7f458806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:27.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.918+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f45940ac830 con 0x7f45ac152bf0 2024-03-19T02:59:27.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.918+0000 7f45a9e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 0x7f458806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:27.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.919+0000 7f45a9e00640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 0x7f458806e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f459c001eb0 tx=0x7f459c001710 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:27.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:27.922+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f459404c030 con 0x7f45ac152bf0 2024-03-19T02:59:28.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.157+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45ac06f950 con 0x7f45ac152bf0 2024-03-19T02:59:28.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.158+0000 7f45a2000640 1 -- 172.21.15.92:0/1209755021 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6003 (secure 0 0 0) 0x7f459407aa60 con 0x7f45ac152bf0 2024-03-19T02:59:28.160 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:28.160 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:24.630720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:28.160 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 15 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 msgr2=0x7f458806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 0x7f458806e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f459c001eb0 tx=0x7f459c001710 comp rx=0 tx=0).stop 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 msgr2=0x7f45ac0728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f459402b7d0 tx=0x7f45940046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 shutdown_connections 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f458806bbd0 0x7f458806e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:28.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ac1593d0 0x7f45ac071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:28.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.160+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ac152bf0 0x7f45ac0728e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:28.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.161+0000 7f45b1200640 1 --2- 172.21.15.92:0/1209755021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ac0c6470 0x7f45ac142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:28.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.161+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 >> 172.21.15.92:0/1209755021 conn(0x7f45ac0c8d90 msgr2=0x7f45ac157c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:28.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.161+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 shutdown_connections 2024-03-19T02:59:28.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:28.161+0000 7f45b1200640 1 -- 172.21.15.92:0/1209755021 wait complete. 2024-03-19T02:59:28.179 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:30.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:59:30.021 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:59:30.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:30.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.403+0000 7f1b80400640 1 Processor -- start 2024-03-19T02:59:30.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.403+0000 7f1b80400640 1 -- start start 2024-03-19T02:59:30.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b78152bf0 0x7f1b7806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 0x7f1b780c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.407 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1b780c6c90 con 0x7f1b78152bf0 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1b78159910 con 0x7f1b781593d0 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b80400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7806e4a0 con 0x7f1b780c6470 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7e400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 0x7f1b780c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b78152bf0 0x7f1b7806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37050/0 (socket says 172.21.15.92:37050) 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7d000640 1 -- 172.21.15.92:0/4265500807 learned_addr learned my addr 172.21.15.92:0/4265500807 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.405+0000 7f1b7e400640 1 --2- 172.21.15.92:0/4265500807 >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 0x7f1b780c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:45728/0 (socket says 172.21.15.92:45728) 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.406+0000 7f1b7d000640 1 -- 172.21.15.92:0/4265500807 >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 msgr2=0x7f1b780c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.406+0000 7f1b7d000640 1 --2- 172.21.15.92:0/4265500807 >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 0x7f1b780c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.406+0000 7f1b7d000640 1 -- 172.21.15.92:0/4265500807 >> v2:172.21.15.92:3300/0 conn(0x7f1b78152bf0 msgr2=0x7f1b7806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.406+0000 7f1b7d000640 1 --2- 172.21.15.92:0/4265500807 >> v2:172.21.15.92:3300/0 conn(0x7f1b78152bf0 0x7f1b7806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.406+0000 7f1b7d000640 1 -- 172.21.15.92:0/4265500807 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b780c9a80 con 0x7f1b780c6470 2024-03-19T02:59:30.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.407+0000 7f1b7d000640 1 --2- 172.21.15.92:0/4265500807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b780c82f0 tx=0x7f1b6402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ab71cffeccae01 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:30.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.407+0000 7f1b77e00640 1 -- 172.21.15.92:0/4265500807 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6402e070 con 0x7f1b780c6470 2024-03-19T02:59:30.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b77e00640 1 -- 172.21.15.92:0/4265500807 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b64037070 con 0x7f1b780c6470 2024-03-19T02:59:30.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b77e00640 1 -- 172.21.15.92:0/4265500807 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b64031a00 con 0x7f1b780c6470 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 -- 172.21.15.92:0/4265500807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 msgr2=0x7f1b780c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 --2- 172.21.15.92:0/4265500807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b780c82f0 tx=0x7f1b6402b880 comp rx=0 tx=0).stop 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 -- 172.21.15.92:0/4265500807 shutdown_connections 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 --2- 172.21.15.92:0/4265500807 >> v2:172.21.15.159:3300/0 conn(0x7f1b781593d0 0x7f1b780c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 --2- 172.21.15.92:0/4265500807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b780c6470 0x7f1b780c3f80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 --2- 172.21.15.92:0/4265500807 >> v2:172.21.15.92:3300/0 conn(0x7f1b78152bf0 0x7f1b7806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 -- 172.21.15.92:0/4265500807 >> 172.21.15.92:0/4265500807 conn(0x7f1b780c8d90 msgr2=0x7f1b780c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:30.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.408+0000 7f1b80400640 1 -- 172.21.15.92:0/4265500807 shutdown_connections 2024-03-19T02:59:30.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.409+0000 7f1b80400640 1 -- 172.21.15.92:0/4265500807 wait complete. 2024-03-19T02:59:30.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.411+0000 7f1b80400640 1 Processor -- start 2024-03-19T02:59:30.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.411+0000 7f1b80400640 1 -- start start 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 0x7f1b781472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 0x7f1b781e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b780c8930 con 0x7f1b780c6470 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7806ea60 con 0x7f1b781593d0 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b80400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7806e6f0 con 0x7f1b78152bf0 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 0x7f1b781472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b7e400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 0x7f1b781e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37052/0 (socket says 172.21.15.92:37052) 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.412+0000 7f1b7d000640 1 -- 172.21.15.92:0/563348119 learned_addr learned my addr 172.21.15.92:0/563348119 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 -- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 msgr2=0x7f1b780cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 -- 172.21.15.92:0/563348119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 msgr2=0x7f1b781472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:30.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 0x7f1b781472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 -- 172.21.15.92:0/563348119 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b78156b80 con 0x7f1b781593d0 2024-03-19T02:59:30.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7d000640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:30.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7da00640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 0x7f1b781472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:30.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b7e400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 0x7f1b781e1ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000e7f0 tx=0x7f1b7000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70017070 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7000c730 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.413+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7000cb60 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.414+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b780ca2c0 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.414+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b780ca880 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.415+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b780684c0 con 0x7f1b781593d0 2024-03-19T02:59:30.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.416+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1b7001a070 con 0x7f1b781593d0 2024-03-19T02:59:30.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.417+0000 7f1b75600640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 0x7f1b5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:30.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.417+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f1b7008a560 con 0x7f1b781593d0 2024-03-19T02:59:30.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.417+0000 7f1b7da00640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 0x7f1b5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:30.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.418+0000 7f1b7da00640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 0x7f1b5806e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00e6c0 tx=0x7f1b6c00d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:30.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.421+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1b78156b80 con 0x7f1b781593d0 2024-03-19T02:59:30.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.658+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b78150a20 con 0x7f1b781593d0 2024-03-19T02:59:30.660 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.659+0000 7f1b75600640 1 -- 172.21.15.92:0/563348119 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6003 (secure 0 0 0) 0x7f1b700588b0 con 0x7f1b781593d0 2024-03-19T02:59:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:30.661 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:24.630720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:30.661 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 15 2024-03-19T02:59:30.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 msgr2=0x7f1b5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 0x7f1b5806e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00e6c0 tx=0x7f1b6c00d040 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 msgr2=0x7f1b781e1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 0x7f1b781e1ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000e7f0 tx=0x7f1b7000ecb0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 shutdown_connections 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1b5806bbd0 0x7f1b5806e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1b781593d0 0x7f1b781e1ea0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1b78152bf0 0x7f1b780cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 --2- 172.21.15.92:0/563348119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b780c6470 0x7f1b781472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 >> 172.21.15.92:0/563348119 conn(0x7f1b780c8d90 msgr2=0x7f1b780c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 shutdown_connections 2024-03-19T02:59:30.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:30.662+0000 7f1b80400640 1 -- 172.21.15.92:0/563348119 wait complete. 2024-03-19T02:59:30.683 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:32.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.893+0000 7fdd36000640 1 Processor -- start 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.894+0000 7fdd36000640 1 -- start start 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.895+0000 7fdd36000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd30154dc0 0x7fdd30071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.895+0000 7fdd36000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 0x7fdd30071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.895+0000 7fdd36000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3015b740 con 0x7fdd300721b0 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.895+0000 7fdd36000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fdd30072660 con 0x7fdd30154dc0 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.895+0000 7fdd36000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3015ba20 con 0x7fdd300c60d0 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdd30154dc0 0x7fdd30071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdd300721b0 0x7fdd300c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55736/0 (socket says 172.21.15.92:55736) 2024-03-19T02:59:32.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2fe00640 1 -- 172.21.15.92:0/2542817501 learned_addr learned my addr 172.21.15.92:0/2542817501 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd26a00640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 0x7fdd30071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2f400640 1 -- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 msgr2=0x7fdd30071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2f400640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 0x7fdd30071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2f400640 1 -- 172.21.15.92:0/2542817501 >> v1:172.21.15.159:6789/0 conn(0x7fdd300721b0 legacy=0x7fdd300c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd2f400640 1 -- 172.21.15.92:0/2542817501 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd300c8400 con 0x7fdd30154dc0 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.896+0000 7fdd26a00640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 0x7fdd30071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.897+0000 7fdd2f400640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd30154dc0 0x7fdd30071190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a1e41d208a26b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.897+0000 7fdd2ea00640 1 -- 172.21.15.92:0/2542817501 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20031780 con 0x7fdd30154dc0 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd2ea00640 1 -- 172.21.15.92:0/2542817501 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd20037070 con 0x7fdd30154dc0 2024-03-19T02:59:32.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd2ea00640 1 -- 172.21.15.92:0/2542817501 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20031b40 con 0x7fdd30154dc0 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 -- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd30154dc0 msgr2=0x7fdd30071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd30154dc0 0x7fdd30071190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002b820 comp rx=0 tx=0).stop 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 -- 172.21.15.92:0/2542817501 shutdown_connections 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300c60d0 0x7fdd30071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 --2- 172.21.15.92:0/2542817501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd30154dc0 0x7fdd30071190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 -- 172.21.15.92:0/2542817501 >> 172.21.15.92:0/2542817501 conn(0x7fdd300c8d90 msgr2=0x7fdd300c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 -- 172.21.15.92:0/2542817501 shutdown_connections 2024-03-19T02:59:32.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.898+0000 7fdd36000640 1 -- 172.21.15.92:0/2542817501 wait complete. 2024-03-19T02:59:32.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.900+0000 7fdd36000640 1 Processor -- start 2024-03-19T02:59:32.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.900+0000 7fdd36000640 1 -- start start 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd300c60d0 0x7fdd300cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30154dc0 0x7fdd300cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300c73a0 con 0x7fdd30154dc0 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301551a0 con 0x7fdd300c60d0 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.901+0000 7fdd36000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdd300c7520 con 0x7fdd300721b0 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd26a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd300c60d0 0x7fdd300cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:37060/0 (socket says 172.21.15.92:37060) 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 -- 172.21.15.92:0/1749373856 learned_addr learned my addr 172.21.15.92:0/1749373856 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2fe00640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30154dc0 0x7fdd300cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 -- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd300c60d0 msgr2=0x7fdd300cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd300c60d0 0x7fdd300cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 -- 172.21.15.92:0/1749373856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30154dc0 msgr2=0x7fdd300cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30154dc0 0x7fdd300cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 -- 172.21.15.92:0/1749373856 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3015b3f0 con 0x7fdd300721b0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.902+0000 7fdd2f400640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdd2002b860 tx=0x7fdd200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.903+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20032610 con 0x7fdd300721b0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.903+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd200327d0 con 0x7fdd300721b0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.903+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20032bc0 con 0x7fdd300721b0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.903+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3014eb20 con 0x7fdd300721b0 2024-03-19T02:59:32.905 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.903+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3014d3a0 con 0x7fdd300721b0 2024-03-19T02:59:32.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.904+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd30073000 con 0x7fdd300721b0 2024-03-19T02:59:32.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.905+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdd2004c060 con 0x7fdd300721b0 2024-03-19T02:59:32.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.906+0000 7fdd27e00640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 0x7fdd0c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:32.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.906+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7fdd200ac440 con 0x7fdd300721b0 2024-03-19T02:59:32.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.906+0000 7fdd26a00640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 0x7fdd0c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:32.908 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.907+0000 7fdd26a00640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 0x7fdd0c06e0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009560 tx=0x7fdd18009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:32.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:32.910+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdd20040070 con 0x7fdd300721b0 2024-03-19T02:59:33.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.132+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd300684c0 con 0x7fdd300721b0 2024-03-19T02:59:33.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.134+0000 7fdd27e00640 1 -- 172.21.15.92:0/1749373856 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6007 (secure 0 0 0) 0x7fdd2002f050 con 0x7fdd300721b0 2024-03-19T02:59:33.135 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:33.135 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:31.981370+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:33.136 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 16 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 msgr2=0x7fdd0c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 0x7fdd0c06e0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009560 tx=0x7fdd18009190 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 msgr2=0x7fdd30158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdd2002b860 tx=0x7fdd200046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 shutdown_connections 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdd0c06bc00 0x7fdd0c06e0c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30154dc0 0x7fdd300cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd300c60d0 0x7fdd300cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 --2- 172.21.15.92:0/1749373856 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd300721b0 0x7fdd30158f40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 >> 172.21.15.92:0/1749373856 conn(0x7fdd300c8d90 msgr2=0x7fdd3006e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 shutdown_connections 2024-03-19T02:59:33.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:33.136+0000 7fdd36000640 1 -- 172.21.15.92:0/1749373856 wait complete. 2024-03-19T02:59:33.156 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:35.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:35.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.371+0000 7f0e2da00640 1 Processor -- start 2024-03-19T02:59:35.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.371+0000 7f0e2da00640 1 -- start start 2024-03-19T02:59:35.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 0x7f0e280c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e2806f800 0x7f0e28072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e28159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0e28159780 con 0x7f0e2806f800 2024-03-19T02:59:35.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f0e280c6c90 con 0x7f0e28071da0 2024-03-19T02:59:35.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e2da00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0e280c8200 con 0x7f0e28158db0 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e26000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e2806f800 0x7f0e28072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e26000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e2806f800 0x7f0e28072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:35596/0 (socket says 172.21.15.92:35596) 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e26000640 1 -- 172.21.15.92:0/1427280738 learned_addr learned my addr 172.21.15.92:0/1427280738 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e26a00640 1 --2- 172.21.15.92:0/1427280738 >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 0x7f0e280c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.373+0000 7f0e27400640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e28159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26000640 1 -- 172.21.15.92:0/1427280738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 msgr2=0x7f0e28159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26000640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e28159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26000640 1 -- 172.21.15.92:0/1427280738 >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 msgr2=0x7f0e280c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26000640 1 --2- 172.21.15.92:0/1427280738 >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 0x7f0e280c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26000640 1 -- 172.21.15.92:0/1427280738 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e280c84e0 con 0x7f0e2806f800 2024-03-19T02:59:35.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.374+0000 7f0e26a00640 1 --2- 172.21.15.92:0/1427280738 >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 0x7f0e280c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.375+0000 7f0e26000640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28072180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0e28075650 tx=0x7f0e1002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4e86c27da92c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.375+0000 7f0e25600640 1 -- 172.21.15.92:0/1427280738 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1002ca80 con 0x7f0e2806f800 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.375+0000 7f0e25600640 1 -- 172.21.15.92:0/1427280738 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1002cc20 con 0x7f0e2806f800 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.375+0000 7f0e25600640 1 -- 172.21.15.92:0/1427280738 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e100343f0 con 0x7f0e2806f800 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 -- 172.21.15.92:0/1427280738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 msgr2=0x7f0e28072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28072180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0e28075650 tx=0x7f0e1002bee0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 -- 172.21.15.92:0/1427280738 shutdown_connections 2024-03-19T02:59:35.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e28159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 --2- 172.21.15.92:0/1427280738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28072180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 --2- 172.21.15.92:0/1427280738 >> v2:172.21.15.159:3300/0 conn(0x7f0e28071da0 0x7f0e280c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 -- 172.21.15.92:0/1427280738 >> 172.21.15.92:0/1427280738 conn(0x7f0e280c8d90 msgr2=0x7f0e280c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 -- 172.21.15.92:0/1427280738 shutdown_connections 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.376+0000 7f0e2da00640 1 -- 172.21.15.92:0/1427280738 wait complete. 2024-03-19T02:59:35.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.378+0000 7f0e2da00640 1 Processor -- start 2024-03-19T02:59:35.380 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.378+0000 7f0e2da00640 1 -- start start 2024-03-19T02:59:35.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 0x7f0e2814e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e280c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0e280726c0 con 0x7f0e2806f800 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2815fde0 con 0x7f0e28071da0 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e2da00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0e28072840 con 0x7f0e28158db0 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:35608/0 (socket says 172.21.15.92:35608) 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 -- 172.21.15.92:0/3874819710 learned_addr learned my addr 172.21.15.92:0/3874819710 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26000640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 0x7f0e2814e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e27400640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e280c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 -- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 msgr2=0x7f0e280c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e280c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 -- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 msgr2=0x7f0e2814e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 0x7f0e2814e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26a00640 1 -- 172.21.15.92:0/3874819710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e280c65a0 con 0x7f0e2806f800 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.380+0000 7f0e26000640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 0x7f0e2814e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e26a00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e9b0 tx=0x7f0e1800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e18017070 con 0x7f0e2806f800 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1800ca00 con 0x7f0e2806f800 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800cdd0 con 0x7f0e2806f800 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e281610e0 con 0x7f0e2806f800 2024-03-19T02:59:35.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.381+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e28160220 con 0x7f0e2806f800 2024-03-19T02:59:35.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.382+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e280684c0 con 0x7f0e2806f800 2024-03-19T02:59:35.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.383+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0e1801a070 con 0x7f0e2806f800 2024-03-19T02:59:35.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.384+0000 7f0e1ea00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 0x7f0e0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:35.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.385+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f0e1808aa80 con 0x7f0e2806f800 2024-03-19T02:59:35.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.385+0000 7f0e26000640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 0x7f0e0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:35.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.386+0000 7f0e26000640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 0x7f0e0406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0e10035000 tx=0x7f0e10030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:35.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.389+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0e18059410 con 0x7f0e2806f800 2024-03-19T02:59:35.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.624+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e28156ae0 con 0x7f0e2806f800 2024-03-19T02:59:35.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.625+0000 7f0e1ea00640 1 -- 172.21.15.92:0/3874819710 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6007 (secure 0 0 0) 0x7f0e18058d10 con 0x7f0e2806f800 2024-03-19T02:59:35.627 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:35.627 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:31.981370+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:35.628 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 16 2024-03-19T02:59:35.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 msgr2=0x7f0e0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:35.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 0x7f0e0406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0e10035000 tx=0x7f0e10030000 comp rx=0 tx=0).stop 2024-03-19T02:59:35.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 msgr2=0x7f0e28155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e9b0 tx=0x7f0e1800ee70 comp rx=0 tx=0).stop 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 shutdown_connections 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0e0406bbd0 0x7f0e0406e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e28158db0 0x7f0e280c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e28071da0 0x7f0e2814e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 --2- 172.21.15.92:0/3874819710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f800 0x7f0e28155d70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 >> 172.21.15.92:0/3874819710 conn(0x7f0e280c8d90 msgr2=0x7f0e2806e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 shutdown_connections 2024-03-19T02:59:35.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:35.629+0000 7f0e2da00640 1 -- 172.21.15.92:0/3874819710 wait complete. 2024-03-19T02:59:35.651 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:37.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:37.866 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.865+0000 7f5589600640 1 Processor -- start 2024-03-19T02:59:37.867 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.865+0000 7f5589600640 1 -- start start 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.866+0000 7f5589600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f55840c5f90 0x7f55840c6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.866+0000 7f5589600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 0x7f558415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.866+0000 7f5589600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55840c80a0 con 0x7f5584066f50 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.866+0000 7f5589600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f55840c8340 con 0x7f55840c5f90 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.866+0000 7f5589600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f55840c8620 con 0x7f5584067350 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5584066f50 0x7f5584072400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:39680/0 (socket says 172.21.15.92:39680) 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582000640 1 -- 172.21.15.92:0/2889966991 learned_addr learned my addr 172.21.15.92:0/2889966991 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582a00640 1 --2- 172.21.15.92:0/2889966991 >> v2:172.21.15.92:3300/0 conn(0x7f55840c5f90 0x7f55840c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5583400640 1 --2- 172.21.15.92:0/2889966991 >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 0x7f558415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582a00640 1 -- 172.21.15.92:0/2889966991 >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 msgr2=0x7f558415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582a00640 1 --2- 172.21.15.92:0/2889966991 >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 0x7f558415e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582a00640 1 -- 172.21.15.92:0/2889966991 >> v1:172.21.15.92:6789/0 conn(0x7f5584066f50 legacy=0x7f5584072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5582a00640 1 -- 172.21.15.92:0/2889966991 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55840c8900 con 0x7f55840c5f90 2024-03-19T02:59:37.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5581600640 1 -- 172.21.15.92:0/2889966991 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895368684 0 0) 0x7f55840c80a0 con 0x7f5584066f50 2024-03-19T02:59:37.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.867+0000 7f5583400640 1 --2- 172.21.15.92:0/2889966991 >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 0x7f558415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.868+0000 7f5582a00640 1 --2- 172.21.15.92:0/2889966991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55840c5f90 0x7f55840c6370 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55840ca790 tx=0x7f557402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62af6ce04b6dc638 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5581600640 1 -- 172.21.15.92:0/2889966991 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574034070 con 0x7f55840c5f90 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5581600640 1 -- 172.21.15.92:0/2889966991 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5574030610 con 0x7f55840c5f90 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5581600640 1 -- 172.21.15.92:0/2889966991 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55740309e0 con 0x7f55840c5f90 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 -- 172.21.15.92:0/2889966991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55840c5f90 msgr2=0x7f55840c6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 --2- 172.21.15.92:0/2889966991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55840c5f90 0x7f55840c6370 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55840ca790 tx=0x7f557402b960 comp rx=0 tx=0).stop 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 -- 172.21.15.92:0/2889966991 shutdown_connections 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 --2- 172.21.15.92:0/2889966991 >> v2:172.21.15.159:3300/0 conn(0x7f5584067350 0x7f558415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 --2- 172.21.15.92:0/2889966991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55840c5f90 0x7f55840c6370 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 -- 172.21.15.92:0/2889966991 >> 172.21.15.92:0/2889966991 conn(0x7f558406f430 msgr2=0x7f558406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:37.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 -- 172.21.15.92:0/2889966991 shutdown_connections 2024-03-19T02:59:37.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.869+0000 7f5589600640 1 -- 172.21.15.92:0/2889966991 wait complete. 2024-03-19T02:59:37.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.871+0000 7f5589600640 1 Processor -- start 2024-03-19T02:59:37.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.871+0000 7f5589600640 1 -- start start 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 0x7f558415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55840c5f90 0x7f558415a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5584071ee0 con 0x7f5584067350 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5584071c00 con 0x7f55840c5f90 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5589600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5584072b10 con 0x7f5584066f50 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5582000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 0x7f558415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.872+0000 7f5582a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36636/0 (socket says 172.21.15.92:36636) 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 -- 172.21.15.92:0/2792376914 learned_addr learned my addr 172.21.15.92:0/2792376914 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5583400640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55840c5f90 0x7f558415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 -- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55840c5f90 msgr2=0x7f558415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55840c5f90 0x7f558415a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 -- 172.21.15.92:0/2792376914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 msgr2=0x7f558415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 0x7f558415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 -- 172.21.15.92:0/2792376914 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55841f1840 con 0x7f5584066f50 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582000640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 0x7f558415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.873+0000 7f5582a00640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5574000c00 tx=0x7f55740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.874+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574034040 con 0x7f5584066f50 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.874+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5574030410 con 0x7f5584066f50 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.874+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574006e10 con 0x7f5584066f50 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.874+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55841f0cc0 con 0x7f5584066f50 2024-03-19T02:59:37.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.874+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55841f1280 con 0x7f5584066f50 2024-03-19T02:59:37.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.875+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5584068de0 con 0x7f5584066f50 2024-03-19T02:59:37.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.876+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5574037070 con 0x7f5584066f50 2024-03-19T02:59:37.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.876+0000 7f557aa00640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 0x7f556406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:37.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.877+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f55740ac540 con 0x7f5584066f50 2024-03-19T02:59:37.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.877+0000 7f5582000640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 0x7f556406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:37.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.877+0000 7f5582000640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 0x7f556406e0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f55841ee540 tx=0x7f5570009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:37.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:37.881+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5584068de0 con 0x7f5584066f50 2024-03-19T02:59:38.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.119+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55840684c0 con 0x7f5584066f50 2024-03-19T02:59:38.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.120+0000 7f557aa00640 1 -- 172.21.15.92:0/2792376914 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6004 (secure 0 0 0) 0x7f557407a890 con 0x7f5584066f50 2024-03-19T02:59:38.122 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:38.122 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:37.051367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:38.123 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 17 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 msgr2=0x7f556406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 0x7f556406e0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f55841ee540 tx=0x7f5570009040 comp rx=0 tx=0).stop 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 msgr2=0x7f55841504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5574000c00 tx=0x7f55740046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 shutdown_connections 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f556406bc00 0x7f556406e0c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55840c5f90 0x7f558415a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:38.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5584067350 0x7f558415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:38.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 --2- 172.21.15.92:0/2792376914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584066f50 0x7f55841504b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:38.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.123+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 >> 172.21.15.92:0/2792376914 conn(0x7f558406f430 msgr2=0x7f5584155640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:38.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.124+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 shutdown_connections 2024-03-19T02:59:38.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:38.124+0000 7f5589600640 1 -- 172.21.15.92:0/2792376914 wait complete. 2024-03-19T02:59:38.143 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4688": { "addr": "172.21.15.159:6839/2444776025", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6838", "nonce": 2444776025, "type": "v2" }, { "addr": "172.21.15.159:6839", "nonce": 2444776025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4688, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T02:59:37.051367+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4688 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T02:59:40.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.363+0000 7f40c7800640 1 Processor -- start 2024-03-19T02:59:40.365 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.363+0000 7f40c7800640 1 -- start start 2024-03-19T02:59:40.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c7800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 0x7f40c0071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:40.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c7800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c015b740 con 0x7f40c0154dc0 2024-03-19T02:59:40.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c7800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c015ba20 con 0x7f40c0072380 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c7800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f40c00c8400 con 0x7f40c00c60d0 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c4e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f40c0154dc0 0x7f40c0071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:39694/0 (socket says 172.21.15.92:39694) 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c4e00640 1 -- 172.21.15.92:0/521719667 learned_addr learned my addr 172.21.15.92:0/521719667 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.365+0000 7f40c5800640 1 --2- 172.21.15.92:0/521719667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 0x7f40c0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.366+0000 7f40c5800640 1 -- 172.21.15.92:0/521719667 >> v1:172.21.15.159:6789/0 conn(0x7f40c0072380 legacy=0x7f40c00c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.366+0000 7f40c5800640 1 -- 172.21.15.92:0/521719667 >> v1:172.21.15.92:6789/0 conn(0x7f40c0154dc0 legacy=0x7f40c0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.366+0000 7f40bf400640 1 -- 172.21.15.92:0/521719667 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3008184707 0 0) 0x7f40c015ba20 con 0x7f40c0072380 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.366+0000 7f40c5800640 1 -- 172.21.15.92:0/521719667 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c00c86e0 con 0x7f40c00c60d0 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.367+0000 7f40c5800640 1 --2- 172.21.15.92:0/521719667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 0x7f40c0071190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009dc0 tx=0x7f40b802bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4966d322d5ff379 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.367+0000 7f40bf400640 1 -- 172.21.15.92:0/521719667 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8039070 con 0x7f40c00c60d0 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.367+0000 7f40bf400640 1 -- 172.21.15.92:0/521719667 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b802c9f0 con 0x7f40c00c60d0 2024-03-19T02:59:40.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.367+0000 7f40bf400640 1 -- 172.21.15.92:0/521719667 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b802cdc0 con 0x7f40c00c60d0 2024-03-19T02:59:40.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 -- 172.21.15.92:0/521719667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 msgr2=0x7f40c0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 --2- 172.21.15.92:0/521719667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 0x7f40c0071190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009dc0 tx=0x7f40b802bcc0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 -- 172.21.15.92:0/521719667 shutdown_connections 2024-03-19T02:59:40.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 --2- 172.21.15.92:0/521719667 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c00c60d0 0x7f40c0071190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 -- 172.21.15.92:0/521719667 >> 172.21.15.92:0/521719667 conn(0x7f40c00c8d90 msgr2=0x7f40c00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:40.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 -- 172.21.15.92:0/521719667 shutdown_connections 2024-03-19T02:59:40.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.368+0000 7f40c7800640 1 -- 172.21.15.92:0/521719667 wait complete. 2024-03-19T02:59:40.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.369+0000 7f40c7800640 1 Processor -- start 2024-03-19T02:59:40.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 -- start start 2024-03-19T02:59:40.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 0x7f40c014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:40.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c00c60d0 0x7f40c00c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 0x7f40c01f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40c00c7520 con 0x7f40c0072380 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f40c00c73a0 con 0x7f40c00c60d0 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.370+0000 7f40c7800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f40c00c76a0 con 0x7f40c0154dc0 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 0x7f40c014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 0x7f40c014dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43388/0 (socket says 172.21.15.92:43388) 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c4e00640 1 -- 172.21.15.92:0/499173010 learned_addr learned my addr 172.21.15.92:0/499173010 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 0x7f40c01f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40bfe00640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c00c60d0 0x7f40c00c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 -- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c00c60d0 msgr2=0x7f40c00c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c00c60d0 0x7f40c00c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 -- 172.21.15.92:0/499173010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 msgr2=0x7f40c014dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 0x7f40c014dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.371+0000 7f40c5800640 1 -- 172.21.15.92:0/499173010 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c01f3f90 con 0x7f40c0154dc0 2024-03-19T02:59:40.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40c5800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 0x7f40c01f3670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40b8033000 tx=0x7f40b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:40.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8044070 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b8034d70 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8039040 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c01f4270 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.372+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c014d220 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.373+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c00684c0 con 0x7f40c0154dc0 2024-03-19T02:59:40.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.374+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f40b802c8b0 con 0x7f40c0154dc0 2024-03-19T02:59:40.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.375+0000 7f40bcc00640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 0x7f40a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:40.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.375+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f40b80ad250 con 0x7f40c0154dc0 2024-03-19T02:59:40.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.375+0000 7f40c4e00640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 0x7f40a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:40.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.376+0000 7f40c4e00640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 0x7f40a406e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f40b4000f60 tx=0x7f40b4000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:40.381 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.380+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f40b807b500 con 0x7f40c0154dc0 2024-03-19T02:59:40.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.625+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40c006f840 con 0x7f40c0154dc0 2024-03-19T02:59:40.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.626+0000 7f40bcc00640 1 -- 172.21.15.92:0/499173010 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6004 (secure 0 0 0) 0x7f40b807f5a0 con 0x7f40c0154dc0 2024-03-19T02:59:40.627 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:40.627 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:37.051367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:40.628 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 17 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 msgr2=0x7f40a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 0x7f40a406e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f40b4000f60 tx=0x7f40b4000bc0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 msgr2=0x7f40c01f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 0x7f40c01f3670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40b8033000 tx=0x7f40b8004a40 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 shutdown_connections 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f40a406bbd0 0x7f40a406e090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40c0154dc0 0x7f40c01f3670 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40c00c60d0 0x7f40c00c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 --2- 172.21.15.92:0/499173010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c0072380 0x7f40c014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 >> 172.21.15.92:0/499173010 conn(0x7f40c00c8d90 msgr2=0x7f40c015a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.628+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 shutdown_connections 2024-03-19T02:59:40.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:40.629+0000 7f40c7800640 1 -- 172.21.15.92:0/499173010 wait complete. 2024-03-19T02:59:40.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:42.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.868+0000 7eff00000640 1 Processor -- start 2024-03-19T02:59:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.868+0000 7eff00000640 1 -- start start 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7eff00000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef8152bf0 0x7efef806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7eff00000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efef80c6470 0x7efef80c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7eff00000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef80c9a80 con 0x7efef81593d0 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7eff00000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7efef80c6c90 con 0x7efef80c6470 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7eff00000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efef8159910 con 0x7efef8152bf0 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7efefe000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efef81593d0 0x7efef80c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:57012/0 (socket says 172.21.15.92:57012) 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7efefe000640 1 -- 172.21.15.92:0/1212386594 learned_addr learned my addr 172.21.15.92:0/1212386594 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7efefcc00640 1 --2- 172.21.15.92:0/1212386594 >> v2:172.21.15.92:3300/0 conn(0x7efef80c6470 0x7efef80c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7efefd600640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef8152bf0 0x7efef806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.870+0000 7efef7e00640 1 -- 172.21.15.92:0/1212386594 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851790373 0 0) 0x7efef80c9a80 con 0x7efef81593d0 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.871+0000 7efef7e00640 1 -- 172.21.15.92:0/1212386594 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee0003520 con 0x7efef81593d0 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.871+0000 7efefcc00640 1 -- 172.21.15.92:0/1212386594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef8152bf0 msgr2=0x7efef806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.871+0000 7efefcc00640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef8152bf0 0x7efef806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.871+0000 7efefcc00640 1 -- 172.21.15.92:0/1212386594 >> v1:172.21.15.159:6789/0 conn(0x7efef81593d0 legacy=0x7efef80c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.871+0000 7efefcc00640 1 -- 172.21.15.92:0/1212386594 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef80c8650 con 0x7efef80c6470 2024-03-19T02:59:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.872+0000 7efefcc00640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef80c6470 0x7efef80c3f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efee40097a0 tx=0x7efee402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d44429823d11252 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.872+0000 7efef7e00640 1 -- 172.21.15.92:0/1212386594 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4031780 con 0x7efef80c6470 2024-03-19T02:59:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.872+0000 7efef7e00640 1 -- 172.21.15.92:0/1212386594 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4037070 con 0x7efef80c6470 2024-03-19T02:59:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.872+0000 7efef7e00640 1 -- 172.21.15.92:0/1212386594 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4031b40 con 0x7efef80c6470 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 -- 172.21.15.92:0/1212386594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef80c6470 msgr2=0x7efef80c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef80c6470 0x7efef80c3f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efee40097a0 tx=0x7efee402b820 comp rx=0 tx=0).stop 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 -- 172.21.15.92:0/1212386594 shutdown_connections 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef80c6470 0x7efef80c3f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 --2- 172.21.15.92:0/1212386594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef8152bf0 0x7efef806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 -- 172.21.15.92:0/1212386594 >> 172.21.15.92:0/1212386594 conn(0x7efef80c8d90 msgr2=0x7efef80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 -- 172.21.15.92:0/1212386594 shutdown_connections 2024-03-19T02:59:42.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.873+0000 7eff00000640 1 -- 172.21.15.92:0/1212386594 wait complete. 2024-03-19T02:59:42.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.875+0000 7eff00000640 1 Processor -- start 2024-03-19T02:59:42.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.875+0000 7eff00000640 1 -- start start 2024-03-19T02:59:42.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efef80c6470 0x7efef81f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef81593d0 0x7efef81ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efef806ea60 con 0x7efef8152bf0 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efef806e4a0 con 0x7efef80c6470 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7eff00000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efef806e6f0 con 0x7efef81593d0 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7efefcc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7efefcc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43410/0 (socket says 172.21.15.92:43410) 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7efefcc00640 1 -- 172.21.15.92:0/1496995893 learned_addr learned my addr 172.21.15.92:0/1496995893 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7efefe000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef81593d0 0x7efef81ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.876+0000 7efefd600640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efef80c6470 0x7efef81f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 -- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef81593d0 msgr2=0x7efef81ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:42.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef81593d0 0x7efef81ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:42.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 -- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efef80c6470 msgr2=0x7efef81f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efef80c6470 0x7efef81f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 -- 172.21.15.92:0/1496995893 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef815a7d0 con 0x7efef8152bf0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efefcc00640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efee4034040 tx=0x7efee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4031ed0 con 0x7efef8152bf0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4032610 con 0x7efef8152bf0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4032a00 con 0x7efef8152bf0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef81f18d0 con 0x7efef8152bf0 2024-03-19T02:59:42.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.877+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef815a4f0 con 0x7efef8152bf0 2024-03-19T02:59:42.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.879+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef80684c0 con 0x7efef8152bf0 2024-03-19T02:59:42.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.880+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efee4032ba0 con 0x7efef8152bf0 2024-03-19T02:59:42.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.880+0000 7efef5600640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 0x7efed806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:42.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.880+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7efee40ac960 con 0x7efef8152bf0 2024-03-19T02:59:42.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.881+0000 7efefd600640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 0x7efed806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:42.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.881+0000 7efefd600640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 0x7efed806e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efeec009b40 tx=0x7efeec009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:42.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:42.885+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efee4040070 con 0x7efef8152bf0 2024-03-19T02:59:43.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.122+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efef80c3b50 con 0x7efef8152bf0 2024-03-19T02:59:43.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.123+0000 7efef5600640 1 -- 172.21.15.92:0/1496995893 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6010 (secure 0 0 0) 0x7efee402f050 con 0x7efef8152bf0 2024-03-19T02:59:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:43.125 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:42.132669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:43.125 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 18 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 msgr2=0x7efed806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 0x7efed806e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efeec009b40 tx=0x7efeec009190 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 msgr2=0x7efef80cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efee4034040 tx=0x7efee40046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 shutdown_connections 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efed806bbd0 0x7efed806e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efef81593d0 0x7efef81ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efef8152bf0 0x7efef80cba40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 --2- 172.21.15.92:0/1496995893 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efef80c6470 0x7efef81f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:43.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.126+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 >> 172.21.15.92:0/1496995893 conn(0x7efef80c8d90 msgr2=0x7efef80c76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:43.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.127+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 shutdown_connections 2024-03-19T02:59:43.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:43.127+0000 7eff00000640 1 -- 172.21.15.92:0/1496995893 wait complete. 2024-03-19T02:59:43.147 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:45.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:45.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.362+0000 7f3a1cc00640 1 Processor -- start 2024-03-19T02:59:45.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.362+0000 7f3a1cc00640 1 -- start start 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.363+0000 7f3a1cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3a180c6470 0x7f3a180c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.363+0000 7f3a1cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a181593d0 0x7f3a180c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a1cc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a180c9a80 con 0x7f3a18152bf0 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a1cc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3a180c6c90 con 0x7f3a180c6470 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a1cc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3a18159910 con 0x7f3a181593d0 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a15e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3a18152bf0 0x7f3a1806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:39698/0 (socket says 172.21.15.92:39698) 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a15e00640 1 -- 172.21.15.92:0/1212565974 learned_addr learned my addr 172.21.15.92:0/1212565974 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a0d400640 1 --2- 172.21.15.92:0/1212565974 >> v2:172.21.15.92:3300/0 conn(0x7f3a180c6470 0x7f3a180c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a16800640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a181593d0 0x7f3a180c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a15400640 1 -- 172.21.15.92:0/1212565974 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2619836996 0 0) 0x7f3a180c9a80 con 0x7f3a18152bf0 2024-03-19T02:59:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a15400640 1 -- 172.21.15.92:0/1212565974 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39fc003520 con 0x7f3a18152bf0 2024-03-19T02:59:45.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a0d400640 1 -- 172.21.15.92:0/1212565974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a181593d0 msgr2=0x7f3a180c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a0d400640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a181593d0 0x7f3a180c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a0d400640 1 -- 172.21.15.92:0/1212565974 >> v1:172.21.15.92:6789/0 conn(0x7f3a18152bf0 legacy=0x7f3a1806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.367 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.364+0000 7f3a0d400640 1 -- 172.21.15.92:0/1212565974 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a180c8650 con 0x7f3a180c6470 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a0d400640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a180c6470 0x7f3a180c3f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a000097a0 tx=0x7f3a0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87de1eae6fea80da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a15400640 1 -- 172.21.15.92:0/1212565974 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000315a0 con 0x7f3a180c6470 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a15400640 1 -- 172.21.15.92:0/1212565974 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a00037070 con 0x7f3a180c6470 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a15400640 1 -- 172.21.15.92:0/1212565974 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00031960 con 0x7f3a180c6470 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 -- 172.21.15.92:0/1212565974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a180c6470 msgr2=0x7f3a180c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a180c6470 0x7f3a180c3f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a000097a0 tx=0x7f3a0002b840 comp rx=0 tx=0).stop 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 -- 172.21.15.92:0/1212565974 shutdown_connections 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a181593d0 0x7f3a180c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/1212565974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a180c6470 0x7f3a180c3f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.368 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.366+0000 7f3a1cc00640 1 -- 172.21.15.92:0/1212565974 >> 172.21.15.92:0/1212565974 conn(0x7f3a180c8d90 msgr2=0x7f3a180c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.367+0000 7f3a1cc00640 1 -- 172.21.15.92:0/1212565974 shutdown_connections 2024-03-19T02:59:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.367+0000 7f3a1cc00640 1 -- 172.21.15.92:0/1212565974 wait complete. 2024-03-19T02:59:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.368+0000 7f3a1cc00640 1 Processor -- start 2024-03-19T02:59:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.368+0000 7f3a1cc00640 1 -- start start 2024-03-19T02:59:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a18152bf0 0x7f3a180c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 0x7f3a18157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1806ea60 con 0x7f3a181593d0 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1806e4a0 con 0x7f3a18152bf0 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.369+0000 7f3a1cc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1806e6f0 con 0x7f3a180c6470 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a15e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 0x7f3a18157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a0d400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a18152bf0 0x7f3a180c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a15e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36704/0 (socket says 172.21.15.92:36704) 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a15e00640 1 -- 172.21.15.92:0/3675201289 learned_addr learned my addr 172.21.15.92:0/3675201289 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 -- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 msgr2=0x7f3a18157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 -- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a18152bf0 msgr2=0x7f3a180c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a18152bf0 0x7f3a180c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 -- 172.21.15.92:0/3675201289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a181f3500 con 0x7f3a181593d0 2024-03-19T02:59:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a15e00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.370+0000 7f3a16800640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 0x7f3a18157de0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000e7c0 tx=0x7f3a1000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.371+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10017070 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.371+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1000c6d0 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.371+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1000cac0 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.371+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1814d0e0 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.371+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1814d6a0 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.372+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a18151f80 con 0x7f3a181593d0 2024-03-19T02:59:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.373+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a1001a070 con 0x7f3a181593d0 2024-03-19T02:59:45.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.374+0000 7f3a0ea00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 0x7f39f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.374+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f3a1008a870 con 0x7f3a181593d0 2024-03-19T02:59:45.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.374+0000 7f3a15e00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 0x7f39f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.375+0000 7f3a15e00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 0x7f39f406e0c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005c00 tx=0x7f3a08009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.380 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.379+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a18151f80 con 0x7f3a181593d0 2024-03-19T02:59:45.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.613+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a180c9950 con 0x7f3a181593d0 2024-03-19T02:59:45.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.615+0000 7f3a0ea00640 1 -- 172.21.15.92:0/3675201289 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6010 (secure 0 0 0) 0x7f3a180c9950 con 0x7f3a181593d0 2024-03-19T02:59:45.617 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:45.617 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:42.132669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:45.617 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 18 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 msgr2=0x7f39f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 0x7f39f406e0c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005c00 tx=0x7f3a08009040 comp rx=0 tx=0).stop 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 msgr2=0x7f3a18157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 0x7f3a18157de0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000e7c0 tx=0x7f3a1000ec80 comp rx=0 tx=0).stop 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 shutdown_connections 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f39f406bc00 0x7f39f406e0c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a181593d0 0x7f3a18157de0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a18152bf0 0x7f3a180c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 --2- 172.21.15.92:0/3675201289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a180c6470 0x7f3a18157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 >> 172.21.15.92:0/3675201289 conn(0x7f3a180c8d90 msgr2=0x7f3a18151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 shutdown_connections 2024-03-19T02:59:45.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:45.618+0000 7f3a1cc00640 1 -- 172.21.15.92:0/3675201289 wait complete. 2024-03-19T02:59:45.640 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:47.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.858+0000 7f6bade00640 1 Processor -- start 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.858+0000 7f6bade00640 1 -- start start 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.859+0000 7f6bade00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ba80c6470 0x7f6ba80c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6bade00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6ba81593d0 0x7f6ba80c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6bade00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba80c9a80 con 0x7f6ba8152bf0 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6bade00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba80c6c90 con 0x7f6ba80c6470 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6bade00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba8159910 con 0x7f6ba81593d0 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6ba6a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6ba8152bf0 0x7f6ba806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:42232/0 (socket says 172.21.15.92:42232) 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6ba6a00640 1 -- 172.21.15.92:0/454486910 learned_addr learned my addr 172.21.15.92:0/454486910 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6b9fe00640 1 --2- 172.21.15.92:0/454486910 >> v2:172.21.15.92:3300/0 conn(0x7f6ba80c6470 0x7f6ba80c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6ba7400640 1 --2- 172.21.15.92:0/454486910 >> v2:172.21.15.159:3300/0 conn(0x7f6ba81593d0 0x7f6ba80c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6ba6000640 1 -- 172.21.15.92:0/454486910 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206565604 0 0) 0x7f6ba80c9a80 con 0x7f6ba8152bf0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6b9fe00640 1 -- 172.21.15.92:0/454486910 >> v2:172.21.15.159:3300/0 conn(0x7f6ba81593d0 msgr2=0x7f6ba80c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6b9fe00640 1 --2- 172.21.15.92:0/454486910 >> v2:172.21.15.159:3300/0 conn(0x7f6ba81593d0 0x7f6ba80c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.860+0000 7f6b9fe00640 1 -- 172.21.15.92:0/454486910 >> v1:172.21.15.92:6789/0 conn(0x7f6ba8152bf0 legacy=0x7f6ba806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.861+0000 7f6b9fe00640 1 -- 172.21.15.92:0/454486910 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba80c8650 con 0x7f6ba80c6470 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6b9fe00640 1 --2- 172.21.15.92:0/454486910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba80c6470 0x7f6ba80c3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80c5070 tx=0x7f6b9002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acf59179a87c890 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6ba6000640 1 -- 172.21.15.92:0/454486910 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900315a0 con 0x7f6ba80c6470 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6ba6000640 1 -- 172.21.15.92:0/454486910 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b90037070 con 0x7f6ba80c6470 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6ba6000640 1 -- 172.21.15.92:0/454486910 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90031960 con 0x7f6ba80c6470 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6bade00640 1 -- 172.21.15.92:0/454486910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba80c6470 msgr2=0x7f6ba80c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6bade00640 1 --2- 172.21.15.92:0/454486910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba80c6470 0x7f6ba80c3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80c5070 tx=0x7f6b9002b840 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6bade00640 1 -- 172.21.15.92:0/454486910 shutdown_connections 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6bade00640 1 --2- 172.21.15.92:0/454486910 >> v2:172.21.15.159:3300/0 conn(0x7f6ba81593d0 0x7f6ba80c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.862+0000 7f6bade00640 1 --2- 172.21.15.92:0/454486910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba80c6470 0x7f6ba80c3f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.863+0000 7f6bade00640 1 -- 172.21.15.92:0/454486910 >> 172.21.15.92:0/454486910 conn(0x7f6ba80c8d90 msgr2=0x7f6ba80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.863+0000 7f6bade00640 1 -- 172.21.15.92:0/454486910 shutdown_connections 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.863+0000 7f6bade00640 1 -- 172.21.15.92:0/454486910 wait complete. 2024-03-19T02:59:47.866 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.864+0000 7f6bade00640 1 Processor -- start 2024-03-19T02:59:47.866 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.864+0000 7f6bade00640 1 -- start start 2024-03-19T02:59:47.868 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ba80c6470 0x7f6ba815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.868 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ba8152bf0 0x7f6ba81f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.868 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.868 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba806ea60 con 0x7f6ba81593d0 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba806e4a0 con 0x7f6ba80c6470 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.865+0000 7f6bade00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba806e6f0 con 0x7f6ba8152bf0 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba6a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ba80c6470 0x7f6ba815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55264/0 (socket says 172.21.15.92:55264) 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 -- 172.21.15.92:0/1042604523 learned_addr learned my addr 172.21.15.92:0/1042604523 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6b9fe00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ba8152bf0 0x7f6ba81f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 -- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ba8152bf0 msgr2=0x7f6ba81f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ba8152bf0 0x7f6ba81f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 -- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ba80c6470 msgr2=0x7f6ba815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ba80c6470 0x7f6ba815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 -- 172.21.15.92:0/1042604523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba8071c40 con 0x7f6ba81593d0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.866+0000 7f6ba7400640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000efc0 tx=0x7f6ba000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.867+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0018020 con 0x7f6ba81593d0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.867+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ba0014480 con 0x7f6ba81593d0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.867+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba8071f00 con 0x7f6ba81593d0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.867+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0009e60 con 0x7f6ba81593d0 2024-03-19T02:59:47.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.867+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba80724c0 con 0x7f6ba81593d0 2024-03-19T02:59:47.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.868+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba80684c0 con 0x7f6ba81593d0 2024-03-19T02:59:47.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.869+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ba0005d10 con 0x7f6ba81593d0 2024-03-19T02:59:47.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.870+0000 7f6b9f400640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.870+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7f6ba008b780 con 0x7f6ba81593d0 2024-03-19T02:59:47.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.870+0000 7f6ba6a00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.871+0000 7f6ba6a00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 0x7f6b8406e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000f60 tx=0x7f6b98007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:47.874+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ba0022030 con 0x7f6ba81593d0 2024-03-19T02:59:48.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.118+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ba814edf0 con 0x7f6ba81593d0 2024-03-19T02:59:48.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.120+0000 7f6b9f400640 1 -- 172.21.15.92:0/1042604523 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6010 (secure 0 0 0) 0x7f6ba0010040 con 0x7f6ba81593d0 2024-03-19T02:59:48.121 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:48.121 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:42.132669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T02:59:48.122 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 18 2024-03-19T02:59:48.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 msgr2=0x7f6b8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:48.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 0x7f6b8406e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000f60 tx=0x7f6b98007d50 comp rx=0 tx=0).stop 2024-03-19T02:59:48.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 msgr2=0x7f6ba81f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:48.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000efc0 tx=0x7f6ba000c490 comp rx=0 tx=0).stop 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 shutdown_connections 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ba81593d0 0x7f6ba81f2590 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ba8152bf0 0x7f6ba81f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 --2- 172.21.15.92:0/1042604523 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ba80c6470 0x7f6ba815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.123+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 >> 172.21.15.92:0/1042604523 conn(0x7f6ba80c8d90 msgr2=0x7f6ba80c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:48.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.124+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 shutdown_connections 2024-03-19T02:59:48.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:48.124+0000 7f6bade00640 1 -- 172.21.15.92:0/1042604523 wait complete. 2024-03-19T02:59:48.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T02:59:50.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:50.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.371+0000 7fd24be00640 1 Processor -- start 2024-03-19T02:59:50.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.371+0000 7fd24be00640 1 -- start start 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd24c0b34c0 0x7fd24c06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c06c520 0x7fd24c0cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24be00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd24c0b3290 con 0x7fd24c154de0 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd24c0cccc0 con 0x7fd24c0b34c0 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24be00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c0b2cf0 con 0x7fd24c06c520 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd24c0b34c0 0x7fd24c06bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd24c0b34c0 0x7fd24c06bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55272/0 (socket says 172.21.15.92:55272) 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24a000640 1 -- 172.21.15.92:0/1427256197 learned_addr learned my addr 172.21.15.92:0/1427256197 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:50.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.373+0000 7fd24b400640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c06c520 0x7fd24c0cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.374+0000 7fd24a000640 1 -- 172.21.15.92:0/1427256197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c06c520 msgr2=0x7fd24c0cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:50.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.374+0000 7fd24a000640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c06c520 0x7fd24c0cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.374+0000 7fd249600640 1 -- 172.21.15.92:0/1427256197 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740665355 0 0) 0x7fd24c0b3290 con 0x7fd24c154de0 2024-03-19T02:59:50.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.374+0000 7fd24a000640 1 -- 172.21.15.92:0/1427256197 >> v1:172.21.15.159:6789/0 conn(0x7fd24c154de0 legacy=0x7fd24c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.374+0000 7fd24a000640 1 -- 172.21.15.92:0/1427256197 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c15ef90 con 0x7fd24c0b34c0 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.375+0000 7fd24a000640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c06bfe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009e50 tx=0x7fd23c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f186abbc3ad2e3df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.375+0000 7fd249600640 1 -- 172.21.15.92:0/1427256197 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c02cb00 con 0x7fd24c0b34c0 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.375+0000 7fd249600640 1 -- 172.21.15.92:0/1427256197 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23c02cca0 con 0x7fd24c0b34c0 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.375+0000 7fd249600640 1 -- 172.21.15.92:0/1427256197 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c034610 con 0x7fd24c0b34c0 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 -- 172.21.15.92:0/1427256197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 msgr2=0x7fd24c06bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c06bfe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009e50 tx=0x7fd23c02bf60 comp rx=0 tx=0).stop 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 -- 172.21.15.92:0/1427256197 shutdown_connections 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c06c520 0x7fd24c0cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 --2- 172.21.15.92:0/1427256197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c06bfe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 -- 172.21.15.92:0/1427256197 >> 172.21.15.92:0/1427256197 conn(0x7fd24c067130 msgr2=0x7fd24c067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 -- 172.21.15.92:0/1427256197 shutdown_connections 2024-03-19T02:59:50.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.376+0000 7fd24be00640 1 -- 172.21.15.92:0/1427256197 wait complete. 2024-03-19T02:59:50.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.378+0000 7fd24be00640 1 Processor -- start 2024-03-19T02:59:50.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.378+0000 7fd24be00640 1 -- start start 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 0x7fd24c15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c154de0 0x7fd24c1f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c15e4b0 con 0x7fd24c0b34c0 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c15e1d0 con 0x7fd24c06c520 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24be00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c15e790 con 0x7fd24c154de0 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24aa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 0x7fd24c15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.382 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55278/0 (socket says 172.21.15.92:55278) 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 -- 172.21.15.92:0/2266477622 learned_addr learned my addr 172.21.15.92:0/2266477622 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24b400640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c154de0 0x7fd24c1f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 -- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c154de0 msgr2=0x7fd24c1f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c154de0 0x7fd24c1f1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 -- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 msgr2=0x7fd24c15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 0x7fd24c15cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24a000640 1 -- 172.21.15.92:0/2266477622 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c0ca0b0 con 0x7fd24c0b34c0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.379+0000 7fd24aa00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 0x7fd24c15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd24a000640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd23c036040 tx=0x7fd23c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c043020 con 0x7fd24c0b34c0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23c02ce50 con 0x7fd24c0b34c0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24c0ca370 con 0x7fd24c0b34c0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c009510 con 0x7fd24c0b34c0 2024-03-19T02:59:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.380+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24c0ca930 con 0x7fd24c0b34c0 2024-03-19T02:59:50.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.381+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24c0b5b40 con 0x7fd24c0b34c0 2024-03-19T02:59:50.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.382+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd23c046020 con 0x7fd24c0b34c0 2024-03-19T02:59:50.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.383+0000 7fd242a00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 0x7fd22806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:50.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.383+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7fd23c0ad0d0 con 0x7fd24c0b34c0 2024-03-19T02:59:50.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.383+0000 7fd24aa00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 0x7fd22806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:50.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.384+0000 7fd24aa00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 0x7fd22806e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd234000f60 tx=0x7fd234009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:50.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.388+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd24c0b5b40 con 0x7fd24c0b34c0 2024-03-19T02:59:50.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.623+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd24c04f580 con 0x7fd24c0b34c0 2024-03-19T02:59:50.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.624+0000 7fd242a00640 1 -- 172.21.15.92:0/2266477622 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6011 (secure 0 0 0) 0x7fd23c04d030 con 0x7fd24c0b34c0 2024-03-19T02:59:50.626 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:50.626 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T02:59:50.626 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 19 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 msgr2=0x7fd22806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 0x7fd22806e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd234000f60 tx=0x7fd234009040 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 msgr2=0x7fd24c1f1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd23c036040 tx=0x7fd23c004a40 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 shutdown_connections 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd22806bbd0 0x7fd22806e090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd24c154de0 0x7fd24c1f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd24c0b34c0 0x7fd24c1f1a80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 --2- 172.21.15.92:0/2266477622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd24c06c520 0x7fd24c15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 >> 172.21.15.92:0/2266477622 conn(0x7fd24c067130 msgr2=0x7fd24c153f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:50.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 shutdown_connections 2024-03-19T02:59:50.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:50.627+0000 7fd24be00640 1 -- 172.21.15.92:0/2266477622 wait complete. 2024-03-19T02:59:50.645 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4261 2024-03-19T02:59:50.646 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2024-03-19T02:59:54.246 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/12' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/10' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/11' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/16' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/14' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/18' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/13' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/139/17' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/139' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/19' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/15' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/20' 2024-03-19T02:59:54.247 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/12' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/10' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/11' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/16' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/14' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/18' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/13' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/102/17' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/102' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/19' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/15' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/20' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/12' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/10' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/11' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/16' 2024-03-19T02:59:54.248 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/14' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/18' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/13' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed './copyofsnap1/test1/129/17' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1/129' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1/test1' 2024-03-19T02:59:54.249 INFO:tasks.workunit.client.0.smithi092.stdout:removed directory './copyofsnap1' 2024-03-19T02:59:54.250 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:59:54.250 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:59:54.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:59:54.353 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-03-19T02:59:54.353 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-19T02:59:54.423 INFO:tasks.workunit.client.0.smithi092.stderr:+ set -e 2024-03-19T02:59:54.423 INFO:tasks.workunit.client.0.smithi092.stderr:+ touch foo 2024-03-19T02:59:54.428 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod +x foo 2024-03-19T02:59:54.431 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir .snap/s 2024-03-19T02:59:54.464 INFO:tasks.workunit.client.0.smithi092.stderr:+ find .snap/s/foo -executable 2024-03-19T02:59:54.464 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep foo 2024-03-19T02:59:54.468 INFO:tasks.workunit.client.0.smithi092.stdout:.snap/s/foo 2024-03-19T02:59:54.469 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir .snap/s 2024-03-19T02:59:54.477 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm foo 2024-03-19T02:59:54.480 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T02:59:54.481 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:59:54.481 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:59:54.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:59:54.568 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-03-19T02:59:54.568 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-19T02:59:54.634 INFO:tasks.workunit.client.0.smithi092.stderr:+ set -e 2024-03-19T02:59:54.635 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir foo 2024-03-19T02:59:54.639 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo x 2024-03-19T02:59:54.640 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:54.873 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir foo/.snap/ss 2024-03-19T02:59:54.903 INFO:tasks.workunit.client.0.smithi092.stderr:+ ln foo/x foo/xx 2024-03-19T02:59:54.907 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat foo/.snap/ss/x 2024-03-19T02:59:54.916 INFO:tasks.workunit.client.0.smithi092.stdout:x 2024-03-19T02:59:54.916 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir foo/.snap/ss 2024-03-19T02:59:54.924 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo a 2024-03-19T02:59:54.925 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo b 2024-03-19T02:59:54.926 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir foo/.snap/s 2024-03-19T02:59:54.960 INFO:tasks.workunit.client.0.smithi092.stderr:++ cat foo/.snap/s/a 2024-03-19T02:59:54.970 INFO:tasks.workunit.client.0.smithi092.stderr:+ r=a 2024-03-19T02:59:54.970 INFO:tasks.workunit.client.0.smithi092.stderr:+ '[' -z a ']' 2024-03-19T02:59:54.970 INFO:tasks.workunit.client.0.smithi092.stderr:+ ln foo/b foo/b2 2024-03-19T02:59:54.974 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat foo/.snap/s/b 2024-03-19T02:59:54.988 INFO:tasks.workunit.client.0.smithi092.stdout:b 2024-03-19T02:59:54.988 INFO:tasks.workunit.client.0.smithi092.stdout:this used to hang: 2024-03-19T02:59:54.989 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 'this used to hang:' 2024-03-19T02:59:54.989 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo more 2024-03-19T02:59:54.989 INFO:tasks.workunit.client.0.smithi092.stdout:oh, it didn't hang! good job. 2024-03-19T02:59:54.989 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-03-19T02:59:54.989 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat foo/b 2024-03-19T02:59:54.993 INFO:tasks.workunit.client.0.smithi092.stdout:b 2024-03-19T02:59:54.994 INFO:tasks.workunit.client.0.smithi092.stdout:more 2024-03-19T02:59:54.994 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir foo/.snap/s 2024-03-19T02:59:55.003 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm -r foo 2024-03-19T02:59:55.024 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:59:55.025 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T02:59:55.026 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:59:55.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:59:55.128 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-03-19T02:59:55.128 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-19T02:59:55.150 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-03-19T02:59:55.149+0000 7f17be65c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T02:59:55.199 INFO:tasks.workunit.client.0.smithi092.stderr:+ set -e 2024-03-19T02:59:55.199 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:55.202 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir .snap/bar 2024-03-19T02:59:55.236 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/foo 2024-03-19T02:59:55.264 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/foo 2024-03-19T02:59:55.273 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:55.277 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/bar/a 2024-03-19T02:59:55.282 INFO:tasks.workunit.client.0.smithi092.stdout: File: .snap/bar/a 2024-03-19T02:59:55.282 INFO:tasks.workunit.client.0.smithi092.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-19T02:59:55.282 INFO:tasks.workunit.client.0.smithi092.stdout:Device: fch/252d Inode: 1099511633797 Links: 2 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout:Access: 2024-03-19 02:59:55.201103126 +0000 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout:Modify: 2024-03-19 02:59:55.201103126 +0000 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout:Change: 2024-03-19 02:59:55.201103126 +0000 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stdout: Birth: 2024-03-19 02:59:55.201103126 +0000 2024-03-19T02:59:55.283 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir .snap/bar 2024-03-19T02:59:55.295 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T02:59:55.295 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T02:59:55.297 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:59:55.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:59:55.389 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-03-19T02:59:55.389 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-19T02:59:55.458 INFO:tasks.workunit.client.0.smithi092.stderr:+ set -e 2024-03-19T02:59:55.459 INFO:tasks.workunit.client.0.smithi092.stderr:++ seq 1 20 2024-03-19T02:59:55.461 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:55.461 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:55.467 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:55.468 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:55.501 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:55.529 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:55.531 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:55.625 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:55.657 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:55.658 INFO:tasks.workunit.client.0.smithi092.stdout: 48252 ? 00:00:00 ps 2024-03-19T02:59:55.658 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:55.679 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:55.684 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:55.694 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:55.694 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:55.700 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:55.700 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:55.736 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:55.767 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:55.771 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:55.829 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:55.852 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:55.852 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:55.852 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:55.853 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:55.853 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:55.853 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:55.853 INFO:tasks.workunit.client.0.smithi092.stdout: 48262 ? 00:00:00 ps 2024-03-19T02:59:55.854 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:55.869 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:55.874 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:55.883 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:55.883 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:55.888 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:55.889 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:55.922 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:55.942 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:55.947 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:56.038 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:56.062 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:56.063 INFO:tasks.workunit.client.0.smithi092.stdout: 48272 ? 00:00:00 ps 2024-03-19T02:59:56.063 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:56.081 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:56.087 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:56.096 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:56.097 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:56.103 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:56.103 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:56.127 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:56.147 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:56.151 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:56.204 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:56.227 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:56.227 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:56.227 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:56.228 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:56.228 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:56.228 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:56.228 INFO:tasks.workunit.client.0.smithi092.stdout: 48282 ? 00:00:00 ps 2024-03-19T02:59:56.229 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:56.249 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:56.254 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:56.264 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:56.264 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:56.270 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:56.270 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:56.292 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:56.311 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:56.314 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:56.372 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:56.395 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:56.396 INFO:tasks.workunit.client.0.smithi092.stdout: 48292 ? 00:00:00 ps 2024-03-19T02:59:56.397 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:56.413 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:56.417 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:56.426 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:56.427 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:56.433 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:56.433 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:56.456 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:56.475 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:56.479 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:56.533 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:56.556 INFO:tasks.workunit.client.0.smithi092.stdout: 48302 ? 00:00:00 ps 2024-03-19T02:59:56.557 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:56.841 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:56.847 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:56.858 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:56.858 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:56.865 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:56.865 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:56.888 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:56.910 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:56.913 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:56.997 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:57.020 INFO:tasks.workunit.client.0.smithi092.stdout: 48312 ? 00:00:00 ps 2024-03-19T02:59:57.021 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:57.041 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:57.046 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:57.057 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:57.057 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:57.064 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:57.064 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:57.091 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:57.111 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:57.114 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:57.172 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:57.195 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:57.196 INFO:tasks.workunit.client.0.smithi092.stdout: 48322 ? 00:00:00 ps 2024-03-19T02:59:57.197 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:57.216 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:57.221 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:57.233 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:57.233 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:57.242 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:57.242 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:57.264 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:57.284 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:57.288 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:57.340 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:57.363 INFO:tasks.workunit.client.0.smithi092.stdout: 48332 ? 00:00:00 ps 2024-03-19T02:59:57.364 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:57.381 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:57.385 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:57.397 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:57.397 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:57.403 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:57.403 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:57.429 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:57.448 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:57.451 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:57.497 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:57.519 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:57.520 INFO:tasks.workunit.client.0.smithi092.stdout: 48342 ? 00:00:00 ps 2024-03-19T02:59:57.521 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:57.649 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-19T02:59:57.650 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-19T02:59:57.650 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-19T02:59:57.652 INFO:tasks.ceph.mds.f:Started 2024-03-19T02:59:57.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:57.726 INFO:tasks.ceph.mds.f.smithi159.stdout:starting mds.f at 2024-03-19T02:59:57.868 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.868+0000 7fe6ffe00640 1 Processor -- start 2024-03-19T02:59:57.869 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.868+0000 7fe6ffe00640 1 -- start start 2024-03-19T02:59:57.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.869+0000 7fe6ffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:57.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.869+0000 7fe6ffe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7000c64f0 con 0x7fe7000cc800 2024-03-19T02:59:57.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6ffe00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe70015e360 con 0x7fe700155100 2024-03-19T02:59:57.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6ffe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe70015e110 con 0x7fe700154d20 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:54230/0 (socket says 172.21.15.92:54230) 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 -- 172.21.15.92:0/254579308 learned_addr learned my addr 172.21.15.92:0/254579308 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 -- 172.21.15.92:0/254579308 >> v1:172.21.15.159:6789/0 conn(0x7fe700155100 legacy=0x7fe7000cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 -- 172.21.15.92:0/254579308 >> v1:172.21.15.92:6789/0 conn(0x7fe7000cc800 legacy=0x7fe70015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fea00640 1 -- 172.21.15.92:0/254579308 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70015e640 con 0x7fe700154d20 2024-03-19T02:59:57.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.870+0000 7fe6fe000640 1 -- 172.21.15.92:0/254579308 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711962185 0 0) 0x7fe70015e360 con 0x7fe700155100 2024-03-19T02:59:57.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6fea00640 1 --2- 172.21.15.92:0/254579308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe70015e290 tx=0x7fe6f002ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d99a2d9dcbec795e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:57.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6fe000640 1 -- 172.21.15.92:0/254579308 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f002d840 con 0x7fe700154d20 2024-03-19T02:59:57.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6fe000640 1 -- 172.21.15.92:0/254579308 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f002d9e0 con 0x7fe700154d20 2024-03-19T02:59:57.872 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6fe000640 1 -- 172.21.15.92:0/254579308 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f002ddb0 con 0x7fe700154d20 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6ffe00640 1 -- 172.21.15.92:0/254579308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 msgr2=0x7fe7000c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/254579308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe70015e290 tx=0x7fe6f002ba80 comp rx=0 tx=0).stop 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.872+0000 7fe6ffe00640 1 -- 172.21.15.92:0/254579308 shutdown_connections 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.873+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/254579308 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7000c5bf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.873+0000 7fe6ffe00640 1 -- 172.21.15.92:0/254579308 >> 172.21.15.92:0/254579308 conn(0x7fe70006db60 msgr2=0x7fe700066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.873+0000 7fe6ffe00640 1 -- 172.21.15.92:0/254579308 shutdown_connections 2024-03-19T02:59:57.873 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.873+0000 7fe6ffe00640 1 -- 172.21.15.92:0/254579308 wait complete. 2024-03-19T02:59:57.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.874+0000 7fe6ffe00640 1 Processor -- start 2024-03-19T02:59:57.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.874+0000 7fe6ffe00640 1 -- start start 2024-03-19T02:59:57.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:57.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe700155100 0x7fe7000ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:57.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 0x7fe7001f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:57.876 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe700067c90 con 0x7fe700155100 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe70015ec00 con 0x7fe7000d05d0 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.875+0000 7fe6ffe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe7000c6130 con 0x7fe700154d20 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:54234/0 (socket says 172.21.15.92:54234) 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 -- 172.21.15.92:0/3150474682 learned_addr learned my addr 172.21.15.92:0/3150474682 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6f7e00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe700155100 0x7fe7000ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6ff400640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 0x7fe7001f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 -- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 msgr2=0x7fe7001f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 0x7fe7001f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 -- 172.21.15.92:0/3150474682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe700155100 msgr2=0x7fe7000ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe700155100 0x7fe7000ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 -- 172.21.15.92:0/3150474682 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7001467b0 con 0x7fe700154d20 2024-03-19T02:59:57.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6ff400640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 0x7fe7001f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:57.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.876+0000 7fe6fea00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002c090 tx=0x7fe6f00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:57.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.877+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00379d0 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.877+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f0047070 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.877+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe700146a90 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.877+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0037dd0 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.877+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7000cfaf0 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.878+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe70004f580 con 0x7fe700154d20 2024-03-19T02:59:57.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.879+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe6f0040050 con 0x7fe700154d20 2024-03-19T02:59:57.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.880+0000 7fe6f6a00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 0x7fe6dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:57.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.880+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6026+0+0 (secure 0 0 0) 0x7fe6f00acea0 con 0x7fe700154d20 2024-03-19T02:59:57.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.880+0000 7fe6f7e00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 0x7fe6dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:57.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.881+0000 7fe6f7e00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 0x7fe6dc06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7001466e0 tx=0x7fe6e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:57.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:57.884+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe6f0050020 con 0x7fe700154d20 2024-03-19T02:59:57.980 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:57.985 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:57.994 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:57.994 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:58.000 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:58.000 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:58.021 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:58.039 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:58.042 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:58.116 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:58.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.137+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7000d09e0 con 0x7fe700154d20 2024-03-19T02:59:58.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.138+0000 7fe6f6a00640 1 -- 172.21.15.92:0/3150474682 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6011 (secure 0 0 0) 0x7fe6f007b0d0 con 0x7fe700154d20 2024-03-19T02:59:58.139 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T02:59:58.139 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T02:59:58.139 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:58.139 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stdout: 48392 ? 00:00:00 ps 2024-03-19T02:59:58.140 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 19 2024-03-19T02:59:58.140 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:58.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 msgr2=0x7fe6dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:58.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 0x7fe6dc06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7001466e0 tx=0x7fe6e8009040 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 msgr2=0x7fe7001461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002c090 tx=0x7fe6f00046c0 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 shutdown_connections 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe6dc06bbd0 0x7fe6dc06e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7000d05d0 0x7fe7001f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe700155100 0x7fe7000ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 --2- 172.21.15.92:0/3150474682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe700154d20 0x7fe7001461a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 >> 172.21.15.92:0/3150474682 conn(0x7fe70006db60 msgr2=0x7fe7000ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 shutdown_connections 2024-03-19T02:59:58.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T02:59:58.141+0000 7fe6ffe00640 1 -- 172.21.15.92:0/3150474682 wait complete. 2024-03-19T02:59:58.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-03-19T02:59:58.677 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:58.682 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:58.693 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:58.693 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:58.698 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:58.698 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:58.724 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:58.744 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:58.747 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:58.844 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:58.867 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:58.868 INFO:tasks.workunit.client.0.smithi092.stdout: 48404 ? 00:00:00 ps 2024-03-19T02:59:58.869 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:58.887 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:58.893 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:58.908 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:58.909 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:58.915 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:58.915 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:58.939 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:58.959 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:58.963 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:59.023 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:59.046 INFO:tasks.workunit.client.0.smithi092.stdout: 48414 ? 00:00:00 ps 2024-03-19T02:59:59.047 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:59.069 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:59.075 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:59.085 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:59.085 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:59.091 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:59.091 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:59.114 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:59.134 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:59.137 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:59.210 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:59.249 INFO:tasks.workunit.client.0.smithi092.stdout: 48424 ? 00:00:00 ps 2024-03-19T02:59:59.250 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:59.330 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:59.335 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:59.358 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:59.359 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:59.366 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:59.367 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:59.436 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:59.486 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:59.489 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T02:59:59.859 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T02:59:59.882 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T02:59:59.882 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T02:59:59.882 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T02:59:59.883 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T02:59:59.883 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T02:59:59.883 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T02:59:59.883 INFO:tasks.workunit.client.0.smithi092.stdout: 48434 ? 00:00:00 ps 2024-03-19T02:59:59.884 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T02:59:59.901 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T02:59:59.907 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T02:59:59.916 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T02:59:59.916 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T02:59:59.922 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T02:59:59.922 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T02:59:59.958 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T02:59:59.989 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T02:59:59.992 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T03:00:00.091 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T03:00:00.114 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T03:00:00.114 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T03:00:00.114 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T03:00:00.115 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T03:00:00.115 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T03:00:00.115 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T03:00:00.115 INFO:tasks.workunit.client.0.smithi092.stdout: 48444 ? 00:00:00 ps 2024-03-19T03:00:00.116 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T03:00:00.135 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T03:00:00.140 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T03:00:00.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:00:00.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:00:00.151 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T03:00:00.151 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T03:00:00.158 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T03:00:00.158 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T03:00:00.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:00.193 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T03:00:00.223 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T03:00:00.225 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T03:00:00.325 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T03:00:00.348 INFO:tasks.workunit.client.0.smithi092.stdout: 48493 ? 00:00:00 ps 2024-03-19T03:00:00.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.346+0000 7fb293e00640 1 Processor -- start 2024-03-19T03:00:00.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.347+0000 7fb293e00640 1 -- start start 2024-03-19T03:00:00.349 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.348+0000 7fb293e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c0cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.348+0000 7fb293e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c15b9e0 con 0x7fb28c15b550 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.348+0000 7fb293e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c0c5340 con 0x7fb28c154b10 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.348+0000 7fb293e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb28c0c5620 con 0x7fb28c0c63e0 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb28c154b10 0x7fb28c15c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:51798/0 (socket says 172.21.15.92:51798) 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c0cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291400640 1 -- 172.21.15.92:0/3283988038 learned_addr learned my addr 172.21.15.92:0/3283988038 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb290a00640 1 --1- 172.21.15.92:0/3283988038 >> v1:172.21.15.92:6789/0 conn(0x7fb28c15b550 0x7fb28c06c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34378/0 (socket says 172.21.15.92:34378) 2024-03-19T03:00:00.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb28be00640 1 -- 172.21.15.92:0/3283988038 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788758619 0 0) 0x7fb28c0c5340 con 0x7fb28c154b10 2024-03-19T03:00:00.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb28be00640 1 -- 172.21.15.92:0/3283988038 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb274003520 con 0x7fb28c154b10 2024-03-19T03:00:00.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291e00640 1 -- 172.21.15.92:0/3283988038 >> v1:172.21.15.159:6789/0 conn(0x7fb28c154b10 legacy=0x7fb28c15c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291e00640 1 -- 172.21.15.92:0/3283988038 >> v1:172.21.15.92:6789/0 conn(0x7fb28c15b550 legacy=0x7fb28c06c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.349+0000 7fb291e00640 1 -- 172.21.15.92:0/3283988038 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28c0c5900 con 0x7fb28c0c63e0 2024-03-19T03:00:00.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.350+0000 7fb291e00640 1 --2- 172.21.15.92:0/3283988038 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c0cae40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb284009dc0 tx=0x7fb28402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22fcce4f7aa6f8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:00.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.351+0000 7fb28be00640 1 -- 172.21.15.92:0/3283988038 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284039070 con 0x7fb28c0c63e0 2024-03-19T03:00:00.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.351+0000 7fb28be00640 1 -- 172.21.15.92:0/3283988038 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28402cd50 con 0x7fb28c0c63e0 2024-03-19T03:00:00.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.351+0000 7fb28be00640 1 -- 172.21.15.92:0/3283988038 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284034660 con 0x7fb28c0c63e0 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 -- 172.21.15.92:0/3283988038 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 msgr2=0x7fb28c0cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 --2- 172.21.15.92:0/3283988038 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c0cae40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb284009dc0 tx=0x7fb28402bfa0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 -- 172.21.15.92:0/3283988038 shutdown_connections 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 --2- 172.21.15.92:0/3283988038 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c0cae40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 -- 172.21.15.92:0/3283988038 >> 172.21.15.92:0/3283988038 conn(0x7fb28c072970 msgr2=0x7fb28c066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:00.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 -- 172.21.15.92:0/3283988038 shutdown_connections 2024-03-19T03:00:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.352+0000 7fb293e00640 1 -- 172.21.15.92:0/3283988038 wait complete. 2024-03-19T03:00:00.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.354+0000 7fb293e00640 1 Processor -- start 2024-03-19T03:00:00.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.354+0000 7fb293e00640 1 -- start start 2024-03-19T03:00:00.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:00.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb28c154b10 0x7fb28c1e4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:00.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb28c0c7770 0x7fb28c0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:00.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb28c067ca0 con 0x7fb28c154b10 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb28c0c5ec0 con 0x7fb28c0c7770 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb293e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb28c154ef0 con 0x7fb28c0c63e0 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb291400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb291400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:54252/0 (socket says 172.21.15.92:54252) 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb291400640 1 -- 172.21.15.92:0/961943542 learned_addr learned my addr 172.21.15.92:0/961943542 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.355+0000 7fb290a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb28c154b10 0x7fb28c1e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb28c0c7770 0x7fb28c0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 -- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb28c0c7770 msgr2=0x7fb28c0c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb28c0c7770 0x7fb28c0c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 -- 172.21.15.92:0/961943542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb28c154b10 msgr2=0x7fb28c1e4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb28c154b10 0x7fb28c1e4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 -- 172.21.15.92:0/961943542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28c0c90a0 con 0x7fb28c0c63e0 2024-03-19T03:00:00.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb291400640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb28000c500 tx=0x7fb28000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:00.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.356+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280017070 con 0x7fb28c0c63e0 2024-03-19T03:00:00.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.357+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb280005ce0 con 0x7fb28c0c63e0 2024-03-19T03:00:00.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.357+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28c0c7d20 con 0x7fb28c0c63e0 2024-03-19T03:00:00.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.357+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280007330 con 0x7fb28c0c63e0 2024-03-19T03:00:00.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.357+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb28c0c82e0 con 0x7fb28c0c63e0 2024-03-19T03:00:00.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.358+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb268005290 con 0x7fb28c0c63e0 2024-03-19T03:00:00.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.359+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb28001a070 con 0x7fb28c0c63e0 2024-03-19T03:00:00.361 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.360+0000 7fb289600640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 0x7fb26c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:00.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.360+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 6082+0+0 (secure 0 0 0) 0x7fb280089f80 con 0x7fb28c0c63e0 2024-03-19T03:00:00.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.360+0000 7fb290a00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 0x7fb26c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:00.362 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T03:00:00.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.361+0000 7fb290a00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 0x7fb26c06e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0c9c80 tx=0x7fb278007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:00.365 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.364+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb280058350 con 0x7fb28c0c63e0 2024-03-19T03:00:00.366 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T03:00:00.376 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T03:00:00.376 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T03:00:00.380 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T03:00:00.380 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T03:00:00.410 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T03:00:00.437 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T03:00:00.439 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T03:00:00.527 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T03:00:00.551 INFO:tasks.workunit.client.0.smithi092.stdout: 48586 ? 00:00:00 ps 2024-03-19T03:00:00.552 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T03:00:00.631 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T03:00:00.634 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T03:00:00.642 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T03:00:00.642 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T03:00:00.648 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T03:00:00.648 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T03:00:00.680 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T03:00:00.708 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T03:00:00.711 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T03:00:00.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.724+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb268005580 con 0x7fb28c0c63e0 2024-03-19T03:00:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.725+0000 7fb289600640 1 -- 172.21.15.92:0/961943542 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7fb28005c3f0 con 0x7fb28c0c63e0 2024-03-19T03:00:00.727 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:00.727 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:00.727 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 msgr2=0x7fb26c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 0x7fb26c06e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0c9c80 tx=0x7fb278007e80 comp rx=0 tx=0).stop 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 msgr2=0x7fb28c1f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb28000c500 tx=0x7fb28000c9c0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 shutdown_connections 2024-03-19T03:00:00.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb26c06bbd0 0x7fb26c06e090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb28c0c7770 0x7fb28c0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb28c154b10 0x7fb28c1e4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 --2- 172.21.15.92:0/961943542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb28c0c63e0 0x7fb28c1f0bc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.727+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 >> 172.21.15.92:0/961943542 conn(0x7fb28c072970 msgr2=0x7fb28c152100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.728+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 shutdown_connections 2024-03-19T03:00:00.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:00.728+0000 7fb293e00640 1 -- 172.21.15.92:0/961943542 wait complete. 2024-03-19T03:00:00.745 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-19T03:00:00.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2024-03-19T03:00:00.776 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T03:00:00.800 INFO:tasks.workunit.client.0.smithi092.stdout: 48598 ? 00:00:00 ps 2024-03-19T03:00:00.801 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T03:00:00.821 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T03:00:00.826 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T03:00:00.834 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 20` 2024-03-19T03:00:00.834 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a 2024-03-19T03:00:00.839 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/one 2024-03-19T03:00:00.839 INFO:tasks.workunit.client.0.smithi092.stderr:+ cat 2024-03-19T03:00:00.870 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir a/.snap/two 2024-03-19T03:00:00.890 INFO:tasks.workunit.client.0.smithi092.stderr:+ chmod 777 a/foo 2024-03-19T03:00:00.893 INFO:tasks.workunit.client.0.smithi092.stderr:+ sync 2024-03-19T03:00:00.951 INFO:tasks.workunit.client.0.smithi092.stderr:+ ps 2024-03-19T03:00:00.975 INFO:tasks.workunit.client.0.smithi092.stdout: PID TTY TIME CMD 2024-03-19T03:00:00.975 INFO:tasks.workunit.client.0.smithi092.stdout: 4307 ? 00:00:00 systemd 2024-03-19T03:00:00.976 INFO:tasks.workunit.client.0.smithi092.stdout: 4309 ? 00:00:00 (sd-pam) 2024-03-19T03:00:00.976 INFO:tasks.workunit.client.0.smithi092.stdout: 22933 ? 00:00:02 sshd 2024-03-19T03:00:00.976 INFO:tasks.workunit.client.0.smithi092.stdout: 48226 ? 00:00:00 timeout 2024-03-19T03:00:00.976 INFO:tasks.workunit.client.0.smithi092.stdout: 48244 ? 00:00:00 snaptest-double 2024-03-19T03:00:00.976 INFO:tasks.workunit.client.0.smithi092.stdout: 48608 ? 00:00:00 ps 2024-03-19T03:00:00.977 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-19T03:00:00.998 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm a/foo 2024-03-19T03:00:01.005 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir a 2024-03-19T03:00:01.015 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T03:00:01.015 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T03:00:01.016 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T03:00:01.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T03:00:01.158 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-03-19T03:00:01.159 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-19T03:00:01.229 INFO:tasks.workunit.client.0.smithi092.stderr:+ mkdir .snap/foo 2024-03-19T03:00:01.248 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-03-19T03:00:01.248 INFO:tasks.workunit.client.0.smithi092.stdout:We want ENOENT, not ESTALE, here. 2024-03-19T03:00:01.249 INFO:tasks.workunit.client.0.smithi092.stderr:++ seq 1 100 2024-03-19T03:00:01.251 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.252 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/1 2024-03-19T03:00:01.253 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.256 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-03-19T03:00:01.257 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.257 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/2 2024-03-19T03:00:01.258 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.261 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-03-19T03:00:01.261 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.262 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/3 2024-03-19T03:00:01.262 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.265 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-03-19T03:00:01.266 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.266 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/4 2024-03-19T03:00:01.267 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.270 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-03-19T03:00:01.270 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.271 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/5 2024-03-19T03:00:01.271 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.274 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-03-19T03:00:01.274 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.275 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/6 2024-03-19T03:00:01.276 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.278 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-03-19T03:00:01.279 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.279 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/7 2024-03-19T03:00:01.280 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.283 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-03-19T03:00:01.283 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.284 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/8 2024-03-19T03:00:01.284 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.287 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-03-19T03:00:01.287 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.288 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/9 2024-03-19T03:00:01.288 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.291 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-03-19T03:00:01.291 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.292 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/10 2024-03-19T03:00:01.292 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.295 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-03-19T03:00:01.296 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.296 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/11 2024-03-19T03:00:01.297 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.299 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-03-19T03:00:01.300 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.301 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/12 2024-03-19T03:00:01.301 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.304 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-03-19T03:00:01.304 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.305 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/13 2024-03-19T03:00:01.305 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.308 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-03-19T03:00:01.309 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.309 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/14 2024-03-19T03:00:01.310 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.312 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-03-19T03:00:01.313 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.314 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/15 2024-03-19T03:00:01.314 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.317 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-03-19T03:00:01.317 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.318 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/16 2024-03-19T03:00:01.318 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.321 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-03-19T03:00:01.321 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.322 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/17 2024-03-19T03:00:01.323 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.325 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-03-19T03:00:01.326 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.326 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/18 2024-03-19T03:00:01.327 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.329 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-03-19T03:00:01.330 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.330 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/19 2024-03-19T03:00:01.331 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.334 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-03-19T03:00:01.334 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.338 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/20 2024-03-19T03:00:01.338 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.338 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-03-19T03:00:01.338 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.339 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/21 2024-03-19T03:00:01.339 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.342 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-03-19T03:00:01.342 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.343 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/22 2024-03-19T03:00:01.343 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.346 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-03-19T03:00:01.346 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.347 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/23 2024-03-19T03:00:01.347 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.350 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-03-19T03:00:01.350 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.351 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/24 2024-03-19T03:00:01.352 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.354 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-03-19T03:00:01.355 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.355 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/25 2024-03-19T03:00:01.356 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.359 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-03-19T03:00:01.359 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.360 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.360 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/26 2024-03-19T03:00:01.363 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-03-19T03:00:01.363 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.364 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/27 2024-03-19T03:00:01.364 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.367 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-03-19T03:00:01.368 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.368 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/28 2024-03-19T03:00:01.369 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.371 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-03-19T03:00:01.372 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.372 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/29 2024-03-19T03:00:01.373 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.376 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-03-19T03:00:01.376 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.377 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/30 2024-03-19T03:00:01.377 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.380 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-03-19T03:00:01.380 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.381 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/31 2024-03-19T03:00:01.381 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.384 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-03-19T03:00:01.384 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.385 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/32 2024-03-19T03:00:01.386 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.388 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-03-19T03:00:01.389 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.389 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/33 2024-03-19T03:00:01.390 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.392 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-03-19T03:00:01.393 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.394 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/34 2024-03-19T03:00:01.394 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.397 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-03-19T03:00:01.397 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.398 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/35 2024-03-19T03:00:01.398 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.401 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-03-19T03:00:01.401 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.402 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/36 2024-03-19T03:00:01.402 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.405 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-03-19T03:00:01.405 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.406 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/37 2024-03-19T03:00:01.407 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.409 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-03-19T03:00:01.410 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.410 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/38 2024-03-19T03:00:01.411 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.414 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-03-19T03:00:01.414 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.415 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/39 2024-03-19T03:00:01.415 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.418 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-03-19T03:00:01.418 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.419 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/40 2024-03-19T03:00:01.419 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.422 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-03-19T03:00:01.422 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.423 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/41 2024-03-19T03:00:01.423 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.426 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-03-19T03:00:01.426 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.427 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/42 2024-03-19T03:00:01.427 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.431 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-03-19T03:00:01.431 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.432 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/43 2024-03-19T03:00:01.432 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.435 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-03-19T03:00:01.435 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.436 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/44 2024-03-19T03:00:01.436 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.439 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-03-19T03:00:01.439 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.440 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/45 2024-03-19T03:00:01.440 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.443 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-03-19T03:00:01.444 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.444 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/46 2024-03-19T03:00:01.445 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.448 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-03-19T03:00:01.448 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.449 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/47 2024-03-19T03:00:01.449 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.453 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-03-19T03:00:01.453 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.454 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/48 2024-03-19T03:00:01.454 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.457 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-03-19T03:00:01.457 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.458 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/49 2024-03-19T03:00:01.458 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.461 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-03-19T03:00:01.462 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.462 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/50 2024-03-19T03:00:01.463 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.465 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-03-19T03:00:01.466 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.466 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/51 2024-03-19T03:00:01.467 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.469 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-03-19T03:00:01.470 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.470 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/52 2024-03-19T03:00:01.471 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.473 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-03-19T03:00:01.474 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.475 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/53 2024-03-19T03:00:01.475 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.477 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-03-19T03:00:01.478 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.479 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/54 2024-03-19T03:00:01.479 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.481 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-03-19T03:00:01.482 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.483 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/55 2024-03-19T03:00:01.483 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.485 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-03-19T03:00:01.486 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.487 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/56 2024-03-19T03:00:01.487 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.490 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-03-19T03:00:01.490 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.491 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/57 2024-03-19T03:00:01.492 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.494 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-03-19T03:00:01.495 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.496 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/58 2024-03-19T03:00:01.496 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.498 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-03-19T03:00:01.499 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.500 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/59 2024-03-19T03:00:01.500 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.503 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-03-19T03:00:01.503 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.504 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/60 2024-03-19T03:00:01.504 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.507 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-03-19T03:00:01.507 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.508 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/61 2024-03-19T03:00:01.509 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.511 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-03-19T03:00:01.512 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.512 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/62 2024-03-19T03:00:01.513 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.515 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-03-19T03:00:01.516 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.517 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.517 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/63 2024-03-19T03:00:01.521 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-03-19T03:00:01.522 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.523 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/64 2024-03-19T03:00:01.523 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.526 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-03-19T03:00:01.526 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.527 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/65 2024-03-19T03:00:01.527 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.530 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-03-19T03:00:01.530 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.531 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/66 2024-03-19T03:00:01.531 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.534 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-03-19T03:00:01.535 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.535 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/67 2024-03-19T03:00:01.536 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.539 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-03-19T03:00:01.539 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.540 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/68 2024-03-19T03:00:01.540 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.542 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-03-19T03:00:01.543 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.544 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/69 2024-03-19T03:00:01.544 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.546 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-03-19T03:00:01.547 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.548 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/70 2024-03-19T03:00:01.548 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.551 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-03-19T03:00:01.551 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.552 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/71 2024-03-19T03:00:01.552 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.555 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-03-19T03:00:01.556 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.557 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/72 2024-03-19T03:00:01.557 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.560 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-03-19T03:00:01.560 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.561 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/73 2024-03-19T03:00:01.561 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.564 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-03-19T03:00:01.564 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.565 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/74 2024-03-19T03:00:01.566 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.568 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-03-19T03:00:01.569 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.570 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/75 2024-03-19T03:00:01.570 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.573 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-03-19T03:00:01.573 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.574 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/76 2024-03-19T03:00:01.574 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.577 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-03-19T03:00:01.578 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.578 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/77 2024-03-19T03:00:01.579 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.581 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-03-19T03:00:01.582 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.583 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/78 2024-03-19T03:00:01.583 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.585 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-03-19T03:00:01.586 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.587 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/79 2024-03-19T03:00:01.587 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.590 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-03-19T03:00:01.590 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.591 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/80 2024-03-19T03:00:01.591 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.594 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-03-19T03:00:01.595 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.595 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/81 2024-03-19T03:00:01.596 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.598 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-03-19T03:00:01.599 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.600 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/82 2024-03-19T03:00:01.600 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.603 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-03-19T03:00:01.603 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.604 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/83 2024-03-19T03:00:01.604 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.607 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-03-19T03:00:01.607 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.609 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/84 2024-03-19T03:00:01.609 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.611 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-03-19T03:00:01.612 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.612 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/85 2024-03-19T03:00:01.613 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.615 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-03-19T03:00:01.616 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.617 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/86 2024-03-19T03:00:01.617 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.620 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-03-19T03:00:01.620 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.621 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/87 2024-03-19T03:00:01.621 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.624 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-03-19T03:00:01.624 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.625 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/88 2024-03-19T03:00:01.625 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.628 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-03-19T03:00:01.628 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.629 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/89 2024-03-19T03:00:01.629 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.632 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-03-19T03:00:01.632 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.633 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/90 2024-03-19T03:00:01.633 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.636 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-03-19T03:00:01.636 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.637 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/91 2024-03-19T03:00:01.638 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.640 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-03-19T03:00:01.641 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.642 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/92 2024-03-19T03:00:01.642 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.645 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-03-19T03:00:01.645 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.646 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/93 2024-03-19T03:00:01.646 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.649 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-03-19T03:00:01.649 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.650 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/94 2024-03-19T03:00:01.651 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.653 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-03-19T03:00:01.654 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.654 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/95 2024-03-19T03:00:01.655 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.657 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-03-19T03:00:01.658 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.658 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/96 2024-03-19T03:00:01.659 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.661 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-03-19T03:00:01.662 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.662 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/97 2024-03-19T03:00:01.663 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.665 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-03-19T03:00:01.666 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.666 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/98 2024-03-19T03:00:01.667 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.669 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-03-19T03:00:01.670 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.670 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/99 2024-03-19T03:00:01.671 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.673 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-03-19T03:00:01.674 INFO:tasks.workunit.client.0.smithi092.stderr:+ for f in `seq 1 100` 2024-03-19T03:00:01.674 INFO:tasks.workunit.client.0.smithi092.stderr:+ stat .snap/foo/100 2024-03-19T03:00:01.675 INFO:tasks.workunit.client.0.smithi092.stderr:+ grep 'No such file' 2024-03-19T03:00:01.678 INFO:tasks.workunit.client.0.smithi092.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-03-19T03:00:01.678 INFO:tasks.workunit.client.0.smithi092.stderr:+ rmdir .snap/foo 2024-03-19T03:00:01.686 INFO:tasks.workunit.client.0.smithi092.stdout:OK 2024-03-19T03:00:01.686 INFO:tasks.workunit.client.0.smithi092.stderr:+ echo OK 2024-03-19T03:00:01.687 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T03:00:01.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T03:00:01.777 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-03-19T03:00:01.777 DEBUG:teuthology.orchestra.run.smithi092:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-19T03:00:01.843 INFO:tasks.workunit.client.0.smithi092.stderr:+ set -e 2024-03-19T03:00:01.843 INFO:tasks.workunit.client.0.smithi092.stderr:+ sudo git config --global http.sslVerify false 2024-03-19T03:00:01.890 INFO:tasks.workunit.client.0.smithi092.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-03-19T03:00:01.935 INFO:tasks.workunit.client.0.smithi092.stderr:+ retried=false 2024-03-19T03:00:01.935 INFO:tasks.workunit.client.0.smithi092.stderr:+ trap -- retry EXIT 2024-03-19T03:00:01.935 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm -rf ceph 2024-03-19T03:00:01.937 INFO:tasks.workunit.client.0.smithi092.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-03-19T03:00:01.944 INFO:tasks.workunit.client.0.smithi092.stderr:Cloning into 'ceph'... 2024-03-19T03:00:30.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:00:30.377 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:00:42.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:42.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.964+0000 7faaf1400640 1 Processor -- start 2024-03-19T03:00:42.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.964+0000 7faaf1400640 1 -- start start 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.965+0000 7faaf1400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faaec152bf0 0x7faaec06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.965+0000 7faaf1400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 0x7faaec0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.965+0000 7faaf1400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaec0c9a80 con 0x7faaec0c6470 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.965+0000 7faaf1400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7faaec0c6c90 con 0x7faaec1593d0 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.965+0000 7faaf1400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7faaec159910 con 0x7faaec152bf0 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaea800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faaec152bf0 0x7faaec06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faae3e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7faaec0c6470 0x7faaec0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:48906/0 (socket says 172.21.15.92:48906) 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faae3e00640 1 -- 172.21.15.92:0/121467108 learned_addr learned my addr 172.21.15.92:0/121467108 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaeb200640 1 --2- 172.21.15.92:0/121467108 >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 0x7faaec0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faae9e00640 1 -- 172.21.15.92:0/121467108 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023844599 0 0) 0x7faaec0c9a80 con 0x7faaec0c6470 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faae9e00640 1 -- 172.21.15.92:0/121467108 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faad0003520 con 0x7faaec0c6470 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaea800640 1 -- 172.21.15.92:0/121467108 >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 msgr2=0x7faaec0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:42.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaea800640 1 --2- 172.21.15.92:0/121467108 >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 0x7faaec0c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:42.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaea800640 1 -- 172.21.15.92:0/121467108 >> v1:172.21.15.159:6789/0 conn(0x7faaec0c6470 legacy=0x7faaec0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:42.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaea800640 1 -- 172.21.15.92:0/121467108 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec0c8650 con 0x7faaec152bf0 2024-03-19T03:00:42.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.966+0000 7faaeb200640 1 --2- 172.21.15.92:0/121467108 >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 0x7faaec0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.967+0000 7faaea800640 1 --2- 172.21.15.92:0/121467108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec152bf0 0x7faaec06eef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faaec075180 tx=0x7faadc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f44a6738b3e773 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faae9e00640 1 -- 172.21.15.92:0/121467108 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc0315a0 con 0x7faaec152bf0 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faae9e00640 1 -- 172.21.15.92:0/121467108 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc037070 con 0x7faaec152bf0 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faae9e00640 1 -- 172.21.15.92:0/121467108 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc031960 con 0x7faaec152bf0 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 -- 172.21.15.92:0/121467108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec152bf0 msgr2=0x7faaec06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 --2- 172.21.15.92:0/121467108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec152bf0 0x7faaec06eef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faaec075180 tx=0x7faadc02b840 comp rx=0 tx=0).stop 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 -- 172.21.15.92:0/121467108 shutdown_connections 2024-03-19T03:00:42.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 --2- 172.21.15.92:0/121467108 >> v2:172.21.15.92:3300/0 conn(0x7faaec1593d0 0x7faaec0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 --2- 172.21.15.92:0/121467108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec152bf0 0x7faaec06eef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.968+0000 7faaf1400640 1 -- 172.21.15.92:0/121467108 >> 172.21.15.92:0/121467108 conn(0x7faaec0c8d90 msgr2=0x7faaec0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.969+0000 7faaf1400640 1 -- 172.21.15.92:0/121467108 shutdown_connections 2024-03-19T03:00:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.969+0000 7faaf1400640 1 -- 172.21.15.92:0/121467108 wait complete. 2024-03-19T03:00:42.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.970+0000 7faaf1400640 1 Processor -- start 2024-03-19T03:00:42.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.970+0000 7faaf1400640 1 -- start start 2024-03-19T03:00:42.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faaec152bf0 0x7faaec1483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 0x7faaec1f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faaec06ea60 con 0x7faaec1593d0 2024-03-19T03:00:42.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faaec06e4a0 con 0x7faaec0c6470 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.971+0000 7faaf1400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faaec06e6f0 con 0x7faaec152bf0 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaeb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 0x7faaec1f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faae3e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faaec152bf0 0x7faaec1483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:54902/0 (socket says 172.21.15.92:54902) 2024-03-19T03:00:42.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 -- 172.21.15.92:0/3121517183 learned_addr learned my addr 172.21.15.92:0/3121517183 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:42.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 -- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faaec152bf0 msgr2=0x7faaec1483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:42.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faaec152bf0 0x7faaec1483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:42.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 -- 172.21.15.92:0/3121517183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 msgr2=0x7faaec1f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:42.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 0x7faaec1f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:42.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 -- 172.21.15.92:0/3121517183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec143be0 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaeb200640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 0x7faaec1f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.972+0000 7faaea800640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc009890 tx=0x7faadc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.973+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc032430 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.973+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc0325d0 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.973+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc0329c0 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.973+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaec143e80 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.973+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaec144440 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.974+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaec0684c0 con 0x7faaec0c6470 2024-03-19T03:00:42.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.975+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faadc032b60 con 0x7faaec0c6470 2024-03-19T03:00:42.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.976+0000 7faae2a00640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 0x7faac806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:42.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.976+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7faadc02f030 con 0x7faaec0c6470 2024-03-19T03:00:42.977 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.976+0000 7faae3e00640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 0x7faac806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:42.978 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.977+0000 7faae3e00640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 0x7faac806e0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faaec156650 tx=0x7faad4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:42.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:42.980+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faadc040070 con 0x7faaec0c6470 2024-03-19T03:00:43.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.218+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faaec0c3b50 con 0x7faaec0c6470 2024-03-19T03:00:43.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.219+0000 7faae2a00640 1 -- 172.21.15.92:0/3121517183 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7faadc07b120 con 0x7faaec0c6470 2024-03-19T03:00:43.221 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:43.221 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:43.221 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 msgr2=0x7faac806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 0x7faac806e0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faaec156650 tx=0x7faad4009040 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 msgr2=0x7faaec14f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc009890 tx=0x7faadc0046c0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 shutdown_connections 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faac806bc00 0x7faac806e0c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaec1593d0 0x7faaec1f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faaec152bf0 0x7faaec1483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 --2- 172.21.15.92:0/3121517183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faaec0c6470 0x7faaec14f8b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 >> 172.21.15.92:0/3121517183 conn(0x7faaec0c8d90 msgr2=0x7faaec156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 shutdown_connections 2024-03-19T03:00:43.224 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.222+0000 7faaf1400640 1 -- 172.21.15.92:0/3121517183 wait complete. 2024-03-19T03:00:43.243 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2024-03-19T03:00:43.244 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-19T03:00:43.244 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:00:43.254 INFO:tasks.ceph.mds.d:Stopped 2024-03-19T03:00:43.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:43.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.462+0000 7feea7000640 1 Processor -- start 2024-03-19T03:00:43.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.462+0000 7feea7000640 1 -- start start 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.463+0000 7feea7000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 0x7feea0158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7feea7000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea00c3b80 con 0x7feea0151ce0 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7feea7000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea0072e70 con 0x7feea00c7da0 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7feea7000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feea0158860 con 0x7feea0151900 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7feea5000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7feea00c7da0 0x7feea0162140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:48910/0 (socket says 172.21.15.92:48910) 2024-03-19T03:00:43.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7feea0151ce0 0x7feea00723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36112/0 (socket says 172.21.15.92:36112) 2024-03-19T03:00:43.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7feea5000640 1 -- 172.21.15.92:0/3230489442 learned_addr learned my addr 172.21.15.92:0/3230489442 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9fe00640 1 --2- 172.21.15.92:0/3230489442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 0x7feea0158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341116152 0 0) 0x7feea0072e70 con 0x7feea00c7da0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee84003520 con 0x7feea00c7da0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1929855316 0 0) 0x7feea00c3b80 con 0x7feea0151ce0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.464+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feea0072e70 con 0x7feea0151ce0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.465+0000 7fee9fe00640 1 -- 172.21.15.92:0/3230489442 >> v1:172.21.15.159:6789/0 conn(0x7feea00c7da0 legacy=0x7feea0162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.465+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 81580417 0 0) 0x7fee84003520 con 0x7feea00c7da0 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.465+0000 7fee9fe00640 1 -- 172.21.15.92:0/3230489442 >> v1:172.21.15.92:6789/0 conn(0x7feea0151ce0 legacy=0x7feea00723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.465+0000 7fee9fe00640 1 -- 172.21.15.92:0/3230489442 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea0073150 con 0x7feea0151900 2024-03-19T03:00:43.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.466+0000 7fee9fe00640 1 --2- 172.21.15.92:0/3230489442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 0x7feea0158320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee90009750 tx=0x7fee9002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf0ff45e84ac63b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.466+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee980019f0 con 0x7feea0151900 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.466+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee90037070 con 0x7feea0151900 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.466+0000 7fee9ea00640 1 -- 172.21.15.92:0/3230489442 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900315a0 con 0x7feea0151900 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 -- 172.21.15.92:0/3230489442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 msgr2=0x7feea0158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 --2- 172.21.15.92:0/3230489442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 0x7feea0158320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee90009750 tx=0x7fee9002b840 comp rx=0 tx=0).stop 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 -- 172.21.15.92:0/3230489442 shutdown_connections 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 --2- 172.21.15.92:0/3230489442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151900 0x7feea0158320 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 -- 172.21.15.92:0/3230489442 >> 172.21.15.92:0/3230489442 conn(0x7feea00c8d90 msgr2=0x7feea00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 -- 172.21.15.92:0/3230489442 shutdown_connections 2024-03-19T03:00:43.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.467+0000 7feea7000640 1 -- 172.21.15.92:0/3230489442 wait complete. 2024-03-19T03:00:43.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.469+0000 7feea7000640 1 Processor -- start 2024-03-19T03:00:43.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.469+0000 7feea7000640 1 -- start start 2024-03-19T03:00:43.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:43.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea0151900 0x7feea01f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:43.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151ce0 0x7feea0144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feea00c9af0 con 0x7feea00c7da0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7feea0071eb0 con 0x7feea0151900 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea7000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feea00c9c70 con 0x7feea0151ce0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51786/0 (socket says 172.21.15.92:51786) 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 -- 172.21.15.92:0/2440467368 learned_addr learned my addr 172.21.15.92:0/2440467368 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9f400640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea0151900 0x7feea01f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7feea5000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151ce0 0x7feea0144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 -- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151ce0 msgr2=0x7feea0144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151ce0 0x7feea0144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 -- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea0151900 msgr2=0x7feea01f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea0151900 0x7feea01f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.470+0000 7fee9fe00640 1 -- 172.21.15.92:0/2440467368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea0153220 con 0x7feea00c7da0 2024-03-19T03:00:43.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7fee9fe00640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fee90000c00 tx=0x7fee900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900317a0 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee90031940 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea01f1480 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee90031d30 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.471+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea0152740 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.472+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea00684c0 con 0x7feea00c7da0 2024-03-19T03:00:43.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.473+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fee90047480 con 0x7feea00c7da0 2024-03-19T03:00:43.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.474+0000 7fee97e00640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 0x7fee8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:43.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.474+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7fee9002f030 con 0x7feea00c7da0 2024-03-19T03:00:43.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.474+0000 7fee9f400640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 0x7fee8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:43.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.475+0000 7fee9f400640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 0x7fee8006e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feea01f3bc0 tx=0x7fee8c007cf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:43.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.478+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fee9004c060 con 0x7feea00c7da0 2024-03-19T03:00:43.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.717+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea00c4b80 con 0x7feea00c7da0 2024-03-19T03:00:43.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.718+0000 7fee97e00640 1 -- 172.21.15.92:0/2440467368 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7fee9007b7f0 con 0x7feea00c7da0 2024-03-19T03:00:43.720 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:43.720 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:43.721 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.721+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 msgr2=0x7fee8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.721+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 0x7fee8006e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feea01f3bc0 tx=0x7fee8c007cf0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.721+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 msgr2=0x7feea014d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.721+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fee90000c00 tx=0x7fee900046c0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 shutdown_connections 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee8006bbd0 0x7fee8006e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea0151ce0 0x7feea0144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea0151900 0x7feea01f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 --2- 172.21.15.92:0/2440467368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feea00c7da0 0x7feea014d9e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:43.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 >> 172.21.15.92:0/2440467368 conn(0x7feea00c8d90 msgr2=0x7feea0150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:43.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 shutdown_connections 2024-03-19T03:00:43.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:43.722+0000 7feea7000640 1 -- 172.21.15.92:0/2440467368 wait complete. 2024-03-19T03:00:43.742 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:45.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:45.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.962+0000 7faeccc00640 1 Processor -- start 2024-03-19T03:00:45.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.962+0000 7faeccc00640 1 -- start start 2024-03-19T03:00:45.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.963+0000 7faeccc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faec8152bf0 0x7faec806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:45.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faeccc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faec80c9a80 con 0x7faec80c6470 2024-03-19T03:00:45.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faeccc00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faec80c8650 con 0x7faec81593d0 2024-03-19T03:00:45.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faeccc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7faec80c6c90 con 0x7faec8152bf0 2024-03-19T03:00:45.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faec5400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7faec80c6470 0x7faec80c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:47852/0 (socket says 172.21.15.92:47852) 2024-03-19T03:00:45.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faec5400640 1 -- 172.21.15.92:0/20317572 learned_addr learned my addr 172.21.15.92:0/20317572 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:45.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faec5e00640 1 --2- 172.21.15.92:0/20317572 >> v2:172.21.15.92:3300/0 conn(0x7faec8152bf0 0x7faec806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:45.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.964+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172248207 0 0) 0x7faec80c9a80 con 0x7faec80c6470 2024-03-19T03:00:45.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.965+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeac003520 con 0x7faec80c6470 2024-03-19T03:00:45.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.965+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225540281 0 0) 0x7faec80c8650 con 0x7faec81593d0 2024-03-19T03:00:45.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.965+0000 7faec5e00640 1 -- 172.21.15.92:0/20317572 >> v1:172.21.15.159:6789/0 conn(0x7faec81593d0 legacy=0x7faec80c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:45.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.965+0000 7faec5e00640 1 -- 172.21.15.92:0/20317572 >> v1:172.21.15.92:6789/0 conn(0x7faec80c6470 legacy=0x7faec80c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:45.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.965+0000 7faec5e00640 1 -- 172.21.15.92:0/20317572 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec80c8930 con 0x7faec8152bf0 2024-03-19T03:00:45.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faec5e00640 1 --2- 172.21.15.92:0/20317572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec8152bf0 0x7faec806eef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faeb80097a0 tx=0x7faeb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4129bd133205706 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:45.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8035020 con 0x7faec8152bf0 2024-03-19T03:00:45.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb8031830 con 0x7faec8152bf0 2024-03-19T03:00:45.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faec4a00640 1 -- 172.21.15.92:0/20317572 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8031be0 con 0x7faec8152bf0 2024-03-19T03:00:45.968 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faeccc00640 1 -- 172.21.15.92:0/20317572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec8152bf0 msgr2=0x7faec806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.966+0000 7faeccc00640 1 --2- 172.21.15.92:0/20317572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec8152bf0 0x7faec806eef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faeb80097a0 tx=0x7faeb802b8d0 comp rx=0 tx=0).stop 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.967+0000 7faeccc00640 1 -- 172.21.15.92:0/20317572 shutdown_connections 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.967+0000 7faeccc00640 1 --2- 172.21.15.92:0/20317572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec8152bf0 0x7faec806eef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.967+0000 7faeccc00640 1 -- 172.21.15.92:0/20317572 >> 172.21.15.92:0/20317572 conn(0x7faec80c8d90 msgr2=0x7faec80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.967+0000 7faeccc00640 1 -- 172.21.15.92:0/20317572 shutdown_connections 2024-03-19T03:00:45.969 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.967+0000 7faeccc00640 1 -- 172.21.15.92:0/20317572 wait complete. 2024-03-19T03:00:45.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.968+0000 7faeccc00640 1 Processor -- start 2024-03-19T03:00:45.970 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.968+0000 7faeccc00640 1 -- start start 2024-03-19T03:00:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec8152bf0 0x7faec81f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec81593d0 0x7faec81f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faec806e4a0 con 0x7faec80c6470 2024-03-19T03:00:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faec8159910 con 0x7faec81593d0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.969+0000 7faeccc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faec806ea60 con 0x7faec8152bf0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38570/0 (socket says 172.21.15.92:38570) 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 -- 172.21.15.92:0/2107170154 learned_addr learned my addr 172.21.15.92:0/2107170154 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5400640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec8152bf0 0x7faec81f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec6800640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec81593d0 0x7faec81f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 -- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec8152bf0 msgr2=0x7faec81f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec8152bf0 0x7faec81f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 -- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec81593d0 msgr2=0x7faec81f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec81593d0 0x7faec81f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 -- 172.21.15.92:0/2107170154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec8072c20 con 0x7faec80c6470 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.970+0000 7faec5e00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009890 tx=0x7faeb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.971+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8035040 con 0x7faec80c6470 2024-03-19T03:00:45.973 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.971+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb803a840 con 0x7faec80c6470 2024-03-19T03:00:45.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.971+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb803ac30 con 0x7faec80c6470 2024-03-19T03:00:45.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.971+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec81e30f0 con 0x7faec80c6470 2024-03-19T03:00:45.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.971+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec81f3d50 con 0x7faec80c6470 2024-03-19T03:00:45.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.972+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec8068de0 con 0x7faec80c6470 2024-03-19T03:00:45.974 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.973+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faeb8038020 con 0x7faec80c6470 2024-03-19T03:00:45.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.974+0000 7faebe000640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 0x7faea806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:45.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.974+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7faeb80acca0 con 0x7faec80c6470 2024-03-19T03:00:45.975 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.974+0000 7faec5400640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 0x7faea806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:45.976 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.975+0000 7faec5400640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 0x7faea806e0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faec814fd90 tx=0x7faeb4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:45.980 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:45.979+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faec8068de0 con 0x7faec80c6470 2024-03-19T03:00:46.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.217+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faec8152fd0 con 0x7faec80c6470 2024-03-19T03:00:46.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.218+0000 7faebe000640 1 -- 172.21.15.92:0/2107170154 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7faec8152fd0 con 0x7faec80c6470 2024-03-19T03:00:46.220 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:46.220 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:46.220 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 msgr2=0x7faea806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 0x7faea806e0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faec814fd90 tx=0x7faeb4009040 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 msgr2=0x7faec80cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009890 tx=0x7faeb80046c0 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 shutdown_connections 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faea806bc00 0x7faea806e0c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec81593d0 0x7faec81f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec8152bf0 0x7faec81f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 --2- 172.21.15.92:0/2107170154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faec80c6470 0x7faec80cb530 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 >> 172.21.15.92:0/2107170154 conn(0x7faec80c8d90 msgr2=0x7faec80c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 shutdown_connections 2024-03-19T03:00:46.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:46.221+0000 7faeccc00640 1 -- 172.21.15.92:0/2107170154 wait complete. 2024-03-19T03:00:46.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:48.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:48.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.453+0000 7f592b800640 1 Processor -- start 2024-03-19T03:00:48.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.454+0000 7f592b800640 1 -- start start 2024-03-19T03:00:48.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.455+0000 7f592b800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.455+0000 7f592b800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 0x7f5924071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.455+0000 7f592b800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f592415b740 con 0x7f59240721b0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.455+0000 7f592b800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5924072660 con 0x7f59240c60d0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.455+0000 7f592b800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f592415ba20 con 0x7f5924154dc0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f591be00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 0x7f5924071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43070/0 (socket says 172.21.15.92:43070) 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 -- 172.21.15.92:0/1561924954 learned_addr learned my addr 172.21.15.92:0/1561924954 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5929800640 1 --1- 172.21.15.92:0/1561924954 >> v1:172.21.15.159:6789/0 conn(0x7f59240721b0 0x7f59240c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:41330/0 (socket says 172.21.15.92:41330) 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 -- 172.21.15.92:0/1561924954 >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 msgr2=0x7f5924071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 --2- 172.21.15.92:0/1561924954 >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 0x7f5924071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 -- 172.21.15.92:0/1561924954 >> v1:172.21.15.159:6789/0 conn(0x7f59240721b0 legacy=0x7f59240c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f5928e00640 1 -- 172.21.15.92:0/1561924954 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240c8400 con 0x7f5924154dc0 2024-03-19T03:00:48.458 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.456+0000 7f591be00640 1 --2- 172.21.15.92:0/1561924954 >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 0x7f5924071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.457+0000 7f5928e00640 1 --2- 172.21.15.92:0/1561924954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5914009870 tx=0x7f591402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe8ddd51ad78038 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.457+0000 7f5923e00640 1 -- 172.21.15.92:0/1561924954 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914035020 con 0x7f5924154dc0 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.457+0000 7f5923e00640 1 -- 172.21.15.92:0/1561924954 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5914031640 con 0x7f5924154dc0 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.457+0000 7f5923e00640 1 -- 172.21.15.92:0/1561924954 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591402d040 con 0x7f5924154dc0 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 -- 172.21.15.92:0/1561924954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 msgr2=0x7f5924071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 --2- 172.21.15.92:0/1561924954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5914009870 tx=0x7f591402b8f0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 -- 172.21.15.92:0/1561924954 shutdown_connections 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 --2- 172.21.15.92:0/1561924954 >> v2:172.21.15.159:3300/0 conn(0x7f59240c60d0 0x7f5924071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 --2- 172.21.15.92:0/1561924954 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5924154dc0 0x7f5924071190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 -- 172.21.15.92:0/1561924954 >> 172.21.15.92:0/1561924954 conn(0x7f59240c8d90 msgr2=0x7f59240c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 -- 172.21.15.92:0/1561924954 shutdown_connections 2024-03-19T03:00:48.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.458+0000 7f592b800640 1 -- 172.21.15.92:0/1561924954 wait complete. 2024-03-19T03:00:48.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.460+0000 7f592b800640 1 Processor -- start 2024-03-19T03:00:48.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.460+0000 7f592b800640 1 -- start start 2024-03-19T03:00:48.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59240721b0 0x7f592414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.462 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5924154dc0 0x7f592415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f59240c73a0 con 0x7f5924154dc0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f59241551a0 con 0x7f59240721b0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f592b800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59240c7520 con 0x7f59240c60d0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f5929800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5924154dc0 0x7f592415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f591be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f5928e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59240721b0 0x7f592414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f591be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:43084/0 (socket says 172.21.15.92:43084) 2024-03-19T03:00:48.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.461+0000 7f591be00640 1 -- 172.21.15.92:0/1193097317 learned_addr learned my addr 172.21.15.92:0/1193097317 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 -- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59240721b0 msgr2=0x7f592414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59240721b0 0x7f592414f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 -- 172.21.15.92:0/1193097317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5924154dc0 msgr2=0x7f592415cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5924154dc0 0x7f592415cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 -- 172.21.15.92:0/1193097317 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592415ef10 con 0x7f59240c60d0 2024-03-19T03:00:48.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f591be00640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f590c00ebe0 tx=0x7f590c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c017070 con 0x7f59240c60d0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f590c00cc30 con 0x7f59240c60d0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592414ecf0 con 0x7f59240c60d0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c013470 con 0x7f59240c60d0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.462+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5924150010 con 0x7f59240c60d0 2024-03-19T03:00:48.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.463+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59240684c0 con 0x7f59240c60d0 2024-03-19T03:00:48.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.464+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f590c01a070 con 0x7f59240c60d0 2024-03-19T03:00:48.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.465+0000 7f5921600640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:48.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.465+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7f590c08ac00 con 0x7f59240c60d0 2024-03-19T03:00:48.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.465+0000 7f5928e00640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:48.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.466+0000 7f5928e00640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 0x7f590006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5914036000 tx=0x7f591403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:48.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.470+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f592415ef10 con 0x7f59240c60d0 2024-03-19T03:00:48.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.705+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f592414f490 con 0x7f59240c60d0 2024-03-19T03:00:48.707 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.706+0000 7f5921600640 1 -- 172.21.15.92:0/1193097317 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f590c058e10 con 0x7f59240c60d0 2024-03-19T03:00:48.708 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:48.708 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:48.708 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:48.710 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 msgr2=0x7f590006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.710 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 0x7f590006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5914036000 tx=0x7f591403c040 comp rx=0 tx=0).stop 2024-03-19T03:00:48.710 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 msgr2=0x7f592414e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f590c00ebe0 tx=0x7f590c00c520 comp rx=0 tx=0).stop 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 shutdown_connections 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5924154dc0 0x7f592415cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59240c60d0 0x7f592414e7b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 --2- 172.21.15.92:0/1193097317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59240721b0 0x7f592414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 >> 172.21.15.92:0/1193097317 conn(0x7f59240c8d90 msgr2=0x7f5924152810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 shutdown_connections 2024-03-19T03:00:48.711 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:48.709+0000 7f592b800640 1 -- 172.21.15.92:0/1193097317 wait complete. 2024-03-19T03:00:48.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:50.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:50.954 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.953+0000 7ffae8400640 1 Processor -- start 2024-03-19T03:00:50.955 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.953+0000 7ffae8400640 1 -- start start 2024-03-19T03:00:50.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.954+0000 7ffae8400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae8400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae00c9a80 con 0x7ffae00c6470 2024-03-19T03:00:50.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae8400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae00c8650 con 0x7ffae01593d0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae8400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffae00c6c90 con 0x7ffae0152bf0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae5000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ffae00c6470 0x7ffae00c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:47862/0 (socket says 172.21.15.92:47862) 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae5000640 1 -- 172.21.15.92:0/2680741140 learned_addr learned my addr 172.21.15.92:0/2680741140 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffae5a00640 1 --2- 172.21.15.92:0/2680741140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419791988 0 0) 0x7ffae00c8650 con 0x7ffae01593d0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffac8003520 con 0x7ffae01593d0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215052167 0 0) 0x7ffae00c9a80 con 0x7ffae00c6470 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.955+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae00c8650 con 0x7ffae00c6470 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.956+0000 7ffae5a00640 1 -- 172.21.15.92:0/2680741140 >> v1:172.21.15.159:6789/0 conn(0x7ffae01593d0 legacy=0x7ffae00c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.956+0000 7ffae5a00640 1 -- 172.21.15.92:0/2680741140 >> v1:172.21.15.92:6789/0 conn(0x7ffae00c6470 legacy=0x7ffae00c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.956+0000 7ffae5a00640 1 -- 172.21.15.92:0/2680741140 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae00c8930 con 0x7ffae0152bf0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.957+0000 7ffae5a00640 1 --2- 172.21.15.92:0/2680741140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae006eef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffad40097d0 tx=0x7ffad402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ece03c6539219f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.957+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4035020 con 0x7ffae0152bf0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.957+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad4031800 con 0x7ffae0152bf0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.957+0000 7ffadfe00640 1 -- 172.21.15.92:0/2680741140 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4031bb0 con 0x7ffae0152bf0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.957+0000 7ffae8400640 1 -- 172.21.15.92:0/2680741140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 msgr2=0x7ffae006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 --2- 172.21.15.92:0/2680741140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae006eef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffad40097d0 tx=0x7ffad402b8a0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 -- 172.21.15.92:0/2680741140 shutdown_connections 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 --2- 172.21.15.92:0/2680741140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae006eef0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 -- 172.21.15.92:0/2680741140 >> 172.21.15.92:0/2680741140 conn(0x7ffae00c8d90 msgr2=0x7ffae00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 -- 172.21.15.92:0/2680741140 shutdown_connections 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.958+0000 7ffae8400640 1 -- 172.21.15.92:0/2680741140 wait complete. 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.959+0000 7ffae8400640 1 Processor -- start 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 -- start start 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 0x7ffae00ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae013fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffae006e4a0 con 0x7ffae01593d0 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffae0159910 con 0x7ffae00c6470 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.960+0000 7ffae8400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffae006ea60 con 0x7ffae0152bf0 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae5000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae013fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae5a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 0x7ffae00ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38584/0 (socket says 172.21.15.92:38584) 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 -- 172.21.15.92:0/2743515036 learned_addr learned my addr 172.21.15.92:0/2743515036 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 -- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 msgr2=0x7ffae013fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae013fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 -- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 msgr2=0x7ffae00ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 0x7ffae00ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae6400640 1 -- 172.21.15.92:0/2743515036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae014e1f0 con 0x7ffae01593d0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.961+0000 7ffae5a00640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 0x7ffae00ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffae6400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffad800c710 tx=0x7ffad800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8003b70 con 0x7ffae01593d0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad8003d10 con 0x7ffae01593d0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad800aa30 con 0x7ffae01593d0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae01f0660 con 0x7ffae01593d0 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.962+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae01f0bc0 con 0x7ffae01593d0 2024-03-19T03:00:50.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.963+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae00684c0 con 0x7ffae01593d0 2024-03-19T03:00:50.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.964+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffad800abd0 con 0x7ffae01593d0 2024-03-19T03:00:50.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.965+0000 7ffadd600640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 0x7ffac006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.966 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.965+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7ffad808a870 con 0x7ffae01593d0 2024-03-19T03:00:50.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.965+0000 7ffae5a00640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 0x7ffac006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.967 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.966+0000 7ffae5a00640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 0x7ffac006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffad4036000 tx=0x7ffad4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.971 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:50.970+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ffad800ce20 con 0x7ffae01593d0 2024-03-19T03:00:51.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.203+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffae006f950 con 0x7ffae01593d0 2024-03-19T03:00:51.205 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.204+0000 7ffadd600640 1 -- 172.21.15.92:0/2743515036 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7ffad8058c40 con 0x7ffae01593d0 2024-03-19T03:00:51.206 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:51.206 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:51.206 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 msgr2=0x7ffac006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 0x7ffac006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffad4036000 tx=0x7ffad4008030 comp rx=0 tx=0).stop 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 msgr2=0x7ffae01400e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffad800c710 tx=0x7ffad800cbd0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 shutdown_connections 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffac006bc00 0x7ffac006e0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffae01593d0 0x7ffae01400e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffae0152bf0 0x7ffae013fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.209 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 --2- 172.21.15.92:0/2743515036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffae00c6470 0x7ffae00ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 >> 172.21.15.92:0/2743515036 conn(0x7ffae00c8d90 msgr2=0x7ffae00c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 shutdown_connections 2024-03-19T03:00:51.210 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:51.207+0000 7ffae8400640 1 -- 172.21.15.92:0/2743515036 wait complete. 2024-03-19T03:00:51.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:53.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:53.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.442+0000 7f8474400640 1 Processor -- start 2024-03-19T03:00:53.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.442+0000 7f8474400640 1 -- start start 2024-03-19T03:00:53.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8474400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f846c0c6470 0x7f846c0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8474400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f846c1593d0 0x7f846c0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8474400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f846c0c9a80 con 0x7f846c152bf0 2024-03-19T03:00:53.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8474400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f846c0c6c90 con 0x7f846c0c6470 2024-03-19T03:00:53.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8474400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f846c159910 con 0x7f846c1593d0 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8471000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f846c0c6470 0x7f846c0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8471000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f846c0c6470 0x7f846c0c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38596/0 (socket says 172.21.15.92:38596) 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8471000640 1 -- 172.21.15.92:0/38422524 learned_addr learned my addr 172.21.15.92:0/38422524 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.444+0000 7f8472400640 1 --2- 172.21.15.92:0/38422524 >> v2:172.21.15.159:3300/0 conn(0x7f846c1593d0 0x7f846c0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.445+0000 7f8471000640 1 -- 172.21.15.92:0/38422524 >> v2:172.21.15.159:3300/0 conn(0x7f846c1593d0 msgr2=0x7f846c0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:53.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.445+0000 7f8471000640 1 --2- 172.21.15.92:0/38422524 >> v2:172.21.15.159:3300/0 conn(0x7f846c1593d0 0x7f846c0c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.445+0000 7f8471000640 1 -- 172.21.15.92:0/38422524 >> v1:172.21.15.159:6789/0 conn(0x7f846c152bf0 legacy=0x7f846c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.445+0000 7f8471000640 1 -- 172.21.15.92:0/38422524 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846c0c8650 con 0x7f846c0c6470 2024-03-19T03:00:53.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.446+0000 7f8471000640 1 --2- 172.21.15.92:0/38422524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c0c6470 0x7f846c0c3f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f846c0c5070 tx=0x7f845802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2781223f7fdb10ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:53.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.446+0000 7f846be00640 1 -- 172.21.15.92:0/38422524 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458035020 con 0x7f846c0c6470 2024-03-19T03:00:53.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.446+0000 7f846be00640 1 -- 172.21.15.92:0/38422524 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8458031870 con 0x7f846c0c6470 2024-03-19T03:00:53.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.446+0000 7f846be00640 1 -- 172.21.15.92:0/38422524 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458031c40 con 0x7f846c0c6470 2024-03-19T03:00:53.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 -- 172.21.15.92:0/38422524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c0c6470 msgr2=0x7f846c0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 --2- 172.21.15.92:0/38422524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c0c6470 0x7f846c0c3f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f846c0c5070 tx=0x7f845802b910 comp rx=0 tx=0).stop 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 -- 172.21.15.92:0/38422524 shutdown_connections 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 --2- 172.21.15.92:0/38422524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c0c6470 0x7f846c0c3f80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 --2- 172.21.15.92:0/38422524 >> v2:172.21.15.159:3300/0 conn(0x7f846c1593d0 0x7f846c0c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 -- 172.21.15.92:0/38422524 >> 172.21.15.92:0/38422524 conn(0x7f846c0c8d90 msgr2=0x7f846c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 -- 172.21.15.92:0/38422524 shutdown_connections 2024-03-19T03:00:53.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.447+0000 7f8474400640 1 -- 172.21.15.92:0/38422524 wait complete. 2024-03-19T03:00:53.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.449+0000 7f8474400640 1 Processor -- start 2024-03-19T03:00:53.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.449+0000 7f8474400640 1 -- start start 2024-03-19T03:00:53.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 0x7f846c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c152bf0 0x7f846c0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f846c06ea60 con 0x7f846c1593d0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f846c06e4a0 con 0x7f846c152bf0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8474400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f846c06e6f0 con 0x7f846c0c6470 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38608/0 (socket says 172.21.15.92:38608) 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 -- 172.21.15.92:0/2250760188 learned_addr learned my addr 172.21.15.92:0/2250760188 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8471a00640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 0x7f846c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8471000640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c152bf0 0x7f846c0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 -- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 msgr2=0x7f846c141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 0x7f846c141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 -- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c152bf0 msgr2=0x7f846c0cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c152bf0 0x7f846c0cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8472400640 1 -- 172.21.15.92:0/2250760188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846c1ee7a0 con 0x7f846c1593d0 2024-03-19T03:00:53.453 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.450+0000 7f8471a00640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 0x7f846c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8472400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8464009cb0 tx=0x7f846400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464018020 con 0x7f846c1593d0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8464014480 con 0x7f846c1593d0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846c1e4b50 con 0x7f846c1593d0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464014870 con 0x7f846c1593d0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.451+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f846c1f2190 con 0x7f846c1593d0 2024-03-19T03:00:53.454 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.452+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846c0684c0 con 0x7f846c1593d0 2024-03-19T03:00:53.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.453+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8464005d10 con 0x7f846c1593d0 2024-03-19T03:00:53.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.455+0000 7f8469600640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 0x7f844c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:53.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.455+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7f846408b920 con 0x7f846c1593d0 2024-03-19T03:00:53.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.455+0000 7f8471a00640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 0x7f844c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:53.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.456+0000 7f8471a00640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 0x7f844c06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8460001700 tx=0x7f8460009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:53.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.459+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8464022030 con 0x7f846c1593d0 2024-03-19T03:00:53.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.698+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f846c14f510 con 0x7f846c1593d0 2024-03-19T03:00:53.700 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.699+0000 7f8469600640 1 -- 172.21.15.92:0/2250760188 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f8464010040 con 0x7f846c1593d0 2024-03-19T03:00:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:53.701 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:53.701 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 msgr2=0x7f844c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 0x7f844c06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8460001700 tx=0x7f8460009040 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 msgr2=0x7f846c13fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8464009cb0 tx=0x7f846400c490 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 shutdown_connections 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f844c06bc00 0x7f844c06e0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f846c1593d0 0x7f846c13fb00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f846c152bf0 0x7f846c0cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 --2- 172.21.15.92:0/2250760188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f846c0c6470 0x7f846c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 >> 172.21.15.92:0/2250760188 conn(0x7f846c0c8d90 msgr2=0x7f846c1501a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:53.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.702+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 shutdown_connections 2024-03-19T03:00:53.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:53.703+0000 7f8474400640 1 -- 172.21.15.92:0/2250760188 wait complete. 2024-03-19T03:00:53.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:55.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.937+0000 7f890e800640 1 Processor -- start 2024-03-19T03:00:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.937+0000 7f890e800640 1 -- start start 2024-03-19T03:00:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890e800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f89080c9010 0x7f890806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890e800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89080c9550 con 0x7f89080c6fc0 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890e800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89080c9af0 con 0x7f8908154ea0 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890e800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f89080c9810 con 0x7f89080c9010 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890de00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f89080c6fc0 0x7f8908165400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34176/0 (socket says 172.21.15.92:34176) 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890de00640 1 -- 172.21.15.92:0/2004364901 learned_addr learned my addr 172.21.15.92:0/2004364901 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:55.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.939+0000 7f890ca00640 1 --2- 172.21.15.92:0/2004364901 >> v2:172.21.15.159:3300/0 conn(0x7f89080c9010 0x7f890806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589493928 0 0) 0x7f89080c9af0 con 0x7f8908154ea0 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f0003520 con 0x7f8908154ea0 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522368605 0 0) 0x7f89080c9550 con 0x7f89080c6fc0 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89080c9af0 con 0x7f89080c6fc0 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f890ca00640 1 -- 172.21.15.92:0/2004364901 >> v1:172.21.15.159:6789/0 conn(0x7f8908154ea0 legacy=0x7f89080c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f890ca00640 1 -- 172.21.15.92:0/2004364901 >> v1:172.21.15.92:6789/0 conn(0x7f89080c6fc0 legacy=0x7f8908165400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:55.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.940+0000 7f890ca00640 1 -- 172.21.15.92:0/2004364901 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890806bc50 con 0x7f89080c9010 2024-03-19T03:00:55.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.941+0000 7f890ca00640 1 --2- 172.21.15.92:0/2004364901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c9010 0x7f890806c730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009d90 tx=0x7f88f802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c72dd4f8073dd47c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:55.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.941+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8039070 con 0x7f89080c9010 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.941+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f802cd00 con 0x7f89080c9010 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.941+0000 7f8907e00640 1 -- 172.21.15.92:0/2004364901 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8034660 con 0x7f89080c9010 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 -- 172.21.15.92:0/2004364901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c9010 msgr2=0x7f890806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 --2- 172.21.15.92:0/2004364901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c9010 0x7f890806c730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009d90 tx=0x7f88f802bf50 comp rx=0 tx=0).stop 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 -- 172.21.15.92:0/2004364901 shutdown_connections 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 --2- 172.21.15.92:0/2004364901 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c9010 0x7f890806c730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 -- 172.21.15.92:0/2004364901 >> 172.21.15.92:0/2004364901 conn(0x7f890806df40 msgr2=0x7f890806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 -- 172.21.15.92:0/2004364901 shutdown_connections 2024-03-19T03:00:55.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.942+0000 7f890e800640 1 -- 172.21.15.92:0/2004364901 wait complete. 2024-03-19T03:00:55.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.944+0000 7f890e800640 1 Processor -- start 2024-03-19T03:00:55.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.944+0000 7f890e800640 1 -- start start 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c6fc0 0x7f89081f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8908154ea0 0x7f89080cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f89080c85a0 con 0x7f89080c9010 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f89080c8420 con 0x7f89080c6fc0 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890e800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f890806b240 con 0x7f8908154ea0 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33272/0 (socket says 172.21.15.92:33272) 2024-03-19T03:00:55.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890ca00640 1 -- 172.21.15.92:0/2601131938 learned_addr learned my addr 172.21.15.92:0/2601131938 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890de00640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8908154ea0 0x7f89080cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.945+0000 7f890d400640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c6fc0 0x7f89081f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 -- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8908154ea0 msgr2=0x7f89080cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8908154ea0 0x7f89080cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 -- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c6fc0 msgr2=0x7f89081f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c6fc0 0x7f89081f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 -- 172.21.15.92:0/2601131938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89081f4b10 con 0x7f89080c9010 2024-03-19T03:00:55.948 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890ca00640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88f8037040 tx=0x7f88f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8039050 con 0x7f89080c9010 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f8045070 con 0x7f89080c9010 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8009450 con 0x7f89080c9010 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890815b470 con 0x7f89080c9010 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.946+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89081f0b90 con 0x7f89080c9010 2024-03-19T03:00:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.948+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89080c6d20 con 0x7f89080c9010 2024-03-19T03:00:55.950 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.949+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f88f803e850 con 0x7f89080c9010 2024-03-19T03:00:55.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.949+0000 7f8905600640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 0x7f88ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:55.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.950+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7f88f80ad040 con 0x7f89080c9010 2024-03-19T03:00:55.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.950+0000 7f890d400640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 0x7f88ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:55.952 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.950+0000 7f890d400640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 0x7f88ec06e0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88fc001710 tx=0x7f88fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:55.956 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:55.954+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f89080c6d20 con 0x7f89080c9010 2024-03-19T03:00:56.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.192+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f890806b570 con 0x7f89080c9010 2024-03-19T03:00:56.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.193+0000 7f8905600640 1 -- 172.21.15.92:0/2601131938 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f890806b570 con 0x7f89080c9010 2024-03-19T03:00:56.195 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:56.195 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:56.195 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 msgr2=0x7f88ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 0x7f88ec06e0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88fc001710 tx=0x7f88fc009040 comp rx=0 tx=0).stop 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 msgr2=0x7f890806f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88f8037040 tx=0x7f88f8004a40 comp rx=0 tx=0).stop 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 shutdown_connections 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f88ec06bc00 0x7f88ec06e0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:56.198 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8908154ea0 0x7f89080cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:56.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89080c9010 0x7f890806f430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:56.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 --2- 172.21.15.92:0/2601131938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89080c6fc0 0x7f89081f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:56.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 >> 172.21.15.92:0/2601131938 conn(0x7f890806df40 msgr2=0x7f89081556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:56.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 shutdown_connections 2024-03-19T03:00:56.199 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:56.196+0000 7f890e800640 1 -- 172.21.15.92:0/2601131938 wait complete. 2024-03-19T03:00:56.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:00:58.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:58.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.437+0000 7efe71600640 1 Processor -- start 2024-03-19T03:00:58.439 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.437+0000 7efe71600640 1 -- start start 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe71600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 0x7efe6c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe71600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efe6c15ea70 0x7efe6c15b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe71600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6c15ef60 con 0x7efe6c071c40 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe71600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c0c6bf0 con 0x7efe6c15ea70 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe71600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c072610 con 0x7efe6c154e00 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe6be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 0x7efe6c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe6b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efe6c15ea70 0x7efe6c15b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.441 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe70c00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efe6c071c40 0x7efe6c0c7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52106/0 (socket says 172.21.15.92:52106) 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.439+0000 7efe70c00640 1 -- 172.21.15.92:0/3114640446 learned_addr learned my addr 172.21.15.92:0/3114640446 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.440+0000 7efe6be00640 1 -- 172.21.15.92:0/3114640446 >> v2:172.21.15.92:3300/0 conn(0x7efe6c15ea70 msgr2=0x7efe6c15b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.440+0000 7efe6be00640 1 --2- 172.21.15.92:0/3114640446 >> v2:172.21.15.92:3300/0 conn(0x7efe6c15ea70 0x7efe6c15b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.440+0000 7efe6be00640 1 -- 172.21.15.92:0/3114640446 >> v1:172.21.15.159:6789/0 conn(0x7efe6c071c40 legacy=0x7efe6c0c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.440+0000 7efe6aa00640 1 -- 172.21.15.92:0/3114640446 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101470838 0 0) 0x7efe6c15ef60 con 0x7efe6c071c40 2024-03-19T03:00:58.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.440+0000 7efe6be00640 1 -- 172.21.15.92:0/3114640446 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c0728b0 con 0x7efe6c154e00 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.441+0000 7efe6be00640 1 --2- 172.21.15.92:0/3114640446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 0x7efe6c15ae40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0097a0 tx=0x7efe5c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2118f021541a05a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.441+0000 7efe6aa00640 1 -- 172.21.15.92:0/3114640446 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c034070 con 0x7efe6c154e00 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.441+0000 7efe6aa00640 1 -- 172.21.15.92:0/3114640446 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe5c030610 con 0x7efe6c154e00 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.441+0000 7efe6aa00640 1 -- 172.21.15.92:0/3114640446 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c0309e0 con 0x7efe6c154e00 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 -- 172.21.15.92:0/3114640446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 msgr2=0x7efe6c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 --2- 172.21.15.92:0/3114640446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 0x7efe6c15ae40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0097a0 tx=0x7efe5c02b960 comp rx=0 tx=0).stop 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 -- 172.21.15.92:0/3114640446 shutdown_connections 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 --2- 172.21.15.92:0/3114640446 >> v2:172.21.15.92:3300/0 conn(0x7efe6c15ea70 0x7efe6c15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 --2- 172.21.15.92:0/3114640446 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c154e00 0x7efe6c15ae40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 -- 172.21.15.92:0/3114640446 >> 172.21.15.92:0/3114640446 conn(0x7efe6c0cc1f0 msgr2=0x7efe6c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 -- 172.21.15.92:0/3114640446 shutdown_connections 2024-03-19T03:00:58.444 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.442+0000 7efe71600640 1 -- 172.21.15.92:0/3114640446 wait complete. 2024-03-19T03:00:58.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.444+0000 7efe71600640 1 Processor -- start 2024-03-19T03:00:58.445 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.444+0000 7efe71600640 1 -- start start 2024-03-19T03:00:58.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe6c071c40 0x7efe6c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 0x7efe6c1f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1551e0 con 0x7efe6c154e00 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c0c7120 con 0x7efe6c071c40 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe71600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c071160 con 0x7efe6c15ea70 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe6b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe70c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 0x7efe6c1f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe6be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe6c071c40 0x7efe6c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.445+0000 7efe6b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33282/0 (socket says 172.21.15.92:33282) 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe70c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 0x7efe6c1f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:53184/0 (socket says 172.21.15.92:53184) 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 -- 172.21.15.92:0/1453136499 learned_addr learned my addr 172.21.15.92:0/1453136499 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 -- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 msgr2=0x7efe6c1f0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 0x7efe6c1f0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 -- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe6c071c40 msgr2=0x7efe6c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe6c071c40 0x7efe6c15a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 -- 172.21.15.92:0/1453136499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c151fe0 con 0x7efe6c154e00 2024-03-19T03:00:58.448 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe6b400640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e870 tx=0x7efe5800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.446+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58017070 con 0x7efe6c154e00 2024-03-19T03:00:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.447+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe5800c940 con 0x7efe6c154e00 2024-03-19T03:00:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.447+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5800cd10 con 0x7efe6c154e00 2024-03-19T03:00:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.447+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c1f0440 con 0x7efe6c154e00 2024-03-19T03:00:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.447+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c156d60 con 0x7efe6c154e00 2024-03-19T03:00:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.448+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe6c04f580 con 0x7efe6c154e00 2024-03-19T03:00:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.449+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe5801a070 con 0x7efe6c154e00 2024-03-19T03:00:58.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.449+0000 7efe63e00640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 0x7efe4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:58.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.450+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6054+0+0 (secure 0 0 0) 0x7efe5808a9b0 con 0x7efe6c154e00 2024-03-19T03:00:58.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.450+0000 7efe6be00640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 0x7efe4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:58.452 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.451+0000 7efe6be00640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 0x7efe4c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe5c036000 tx=0x7efe5c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:58.456 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.454+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe58058ce0 con 0x7efe6c154e00 2024-03-19T03:00:58.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.689+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe6c06d740 con 0x7efe6c154e00 2024-03-19T03:00:58.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.690+0000 7efe63e00640 1 -- 172.21.15.92:0/1453136499 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7efe5805cd80 con 0x7efe6c154e00 2024-03-19T03:00:58.692 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:00:58.692 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5041,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T02:59:49.143019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4688":{"gid":4688,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":37,"addr":"172.21.15.159:6839/2444776025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":2444776025},{"type":"v1","addr":"172.21.15.159:6839","nonce":2444776025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4688,"qdb_cluster":[4688]},"id":1}]} 2024-03-19T03:00:58.692 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 20 2024-03-19T03:00:58.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 msgr2=0x7efe4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 0x7efe4c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe5c036000 tx=0x7efe5c03b040 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 msgr2=0x7efe6c1e49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e870 tx=0x7efe5800ed30 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 shutdown_connections 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7efe4c06bbd0 0x7efe4c06e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6c15ea70 0x7efe6c1f0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe6c154e00 0x7efe6c1e49d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 --2- 172.21.15.92:0/1453136499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe6c071c40 0x7efe6c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 >> 172.21.15.92:0/1453136499 conn(0x7efe6c0cc1f0 msgr2=0x7efe6c153860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 shutdown_connections 2024-03-19T03:00:58.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:00:58.693+0000 7efe71600640 1 -- 172.21.15.92:0/1453136499 wait complete. 2024-03-19T03:00:58.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:01:00.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:01:00.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:01:00.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:00.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.932+0000 7f476de00640 1 Processor -- start 2024-03-19T03:01:00.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.933+0000 7f476de00640 1 -- start start 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f47680cd1c0 0x7f47680cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476de00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47680b3290 con 0x7f47680b34c0 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476de00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f47680cccc0 con 0x7f47680cd1c0 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476de00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f47680b2cf0 con 0x7f4768154de0 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f4767e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f47680b34c0 0x7f476806bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52116/0 (socket says 172.21.15.92:52116) 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.934+0000 7f476ca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f4767e00640 1 -- 172.21.15.92:0/3838642433 learned_addr learned my addr 172.21.15.92:0/3838642433 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:00.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f476d400640 1 --2- 172.21.15.92:0/3838642433 >> v2:172.21.15.92:3300/0 conn(0x7f47680cd1c0 0x7f47680cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f4767400640 1 -- 172.21.15.92:0/3838642433 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505678693 0 0) 0x7f47680b3290 con 0x7f47680b34c0 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f4767400640 1 -- 172.21.15.92:0/3838642433 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f474c003520 con 0x7f47680b34c0 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f476ca00640 1 -- 172.21.15.92:0/3838642433 >> v2:172.21.15.92:3300/0 conn(0x7f47680cd1c0 msgr2=0x7f47680cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f476ca00640 1 --2- 172.21.15.92:0/3838642433 >> v2:172.21.15.92:3300/0 conn(0x7f47680cd1c0 0x7f47680cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f476ca00640 1 -- 172.21.15.92:0/3838642433 >> v1:172.21.15.159:6789/0 conn(0x7f47680b34c0 legacy=0x7f476806bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.935+0000 7f476ca00640 1 -- 172.21.15.92:0/3838642433 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476815ef90 con 0x7f4768154de0 2024-03-19T03:01:00.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.936+0000 7f476ca00640 1 --2- 172.21.15.92:0/3838642433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cb170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf25e68ac5f12a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:00.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.936+0000 7f4767400640 1 -- 172.21.15.92:0/3838642433 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750035020 con 0x7f4768154de0 2024-03-19T03:01:00.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.936+0000 7f4767400640 1 -- 172.21.15.92:0/3838642433 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4750031830 con 0x7f4768154de0 2024-03-19T03:01:00.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.936+0000 7f4767400640 1 -- 172.21.15.92:0/3838642433 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750031be0 con 0x7f4768154de0 2024-03-19T03:01:00.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 -- 172.21.15.92:0/3838642433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 msgr2=0x7f47680cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:00.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 --2- 172.21.15.92:0/3838642433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cb170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475002b8d0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 -- 172.21.15.92:0/3838642433 shutdown_connections 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 --2- 172.21.15.92:0/3838642433 >> v2:172.21.15.92:3300/0 conn(0x7f47680cd1c0 0x7f47680cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 --2- 172.21.15.92:0/3838642433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cb170 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 -- 172.21.15.92:0/3838642433 >> 172.21.15.92:0/3838642433 conn(0x7f4768067130 msgr2=0x7f4768067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 -- 172.21.15.92:0/3838642433 shutdown_connections 2024-03-19T03:01:00.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.937+0000 7f476de00640 1 -- 172.21.15.92:0/3838642433 wait complete. 2024-03-19T03:01:00.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.939+0000 7f476de00640 1 Processor -- start 2024-03-19T03:01:00.941 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.939+0000 7f476de00640 1 -- start start 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47680cd1c0 0x7f47680cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f476815e4b0 con 0x7f47680b34c0 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f476815e1d0 con 0x7f47680cd1c0 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476de00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f476815e790 con 0x7f4768154de0 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33300/0 (socket says 172.21.15.92:33300) 2024-03-19T03:01:00.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476ca00640 1 -- 172.21.15.92:0/2912138064 learned_addr learned my addr 172.21.15.92:0/2912138064 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f476d400640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.940+0000 7f4767e00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47680cd1c0 0x7f47680cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 -- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47680cd1c0 msgr2=0x7f47680cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47680cd1c0 0x7f47680cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 -- 172.21.15.92:0/2912138064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 msgr2=0x7f47680b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 -- 172.21.15.92:0/2912138064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680ce350 con 0x7f4768154de0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476ca00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476d400640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cf330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f476000e820 tx=0x7f476000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760017070 con 0x7f4768154de0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476000c940 con 0x7f4768154de0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476000cd10 con 0x7f4768154de0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.941+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47681f0e30 con 0x7f4768154de0 2024-03-19T03:01:00.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.942+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47681f1110 con 0x7f4768154de0 2024-03-19T03:01:00.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.943+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476804f580 con 0x7f4768154de0 2024-03-19T03:01:00.945 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.944+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f476001a070 con 0x7f4768154de0 2024-03-19T03:01:00.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.945+0000 7f4764c00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 0x7f474406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:00.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.945+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f476008aae0 con 0x7f4768154de0 2024-03-19T03:01:00.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.945+0000 7f476ca00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 0x7f474406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:00.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.946+0000 7f476ca00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 0x7f474406e0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:00.951 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:00.950+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4760058d60 con 0x7f4768154de0 2024-03-19T03:01:01.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.187+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4768067920 con 0x7f4768154de0 2024-03-19T03:01:01.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.189+0000 7f4764c00640 1 -- 172.21.15.92:0/2912138064 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6003 (secure 0 0 0) 0x7f476005ce00 con 0x7f4768154de0 2024-03-19T03:01:01.190 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:01.190 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:00:59.656375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:01.191 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 21 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 msgr2=0x7f474406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 0x7f474406e0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475003c040 comp rx=0 tx=0).stop 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 msgr2=0x7f47680cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cf330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f476000e820 tx=0x7f476000ece0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 shutdown_connections 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f474406bc00 0x7f474406e0c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4768154de0 0x7f47680cf330 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47680cd1c0 0x7f47680cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 --2- 172.21.15.92:0/2912138064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47680b34c0 0x7f47680b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 >> 172.21.15.92:0/2912138064 conn(0x7f4768067130 msgr2=0x7f476806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.191+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 shutdown_connections 2024-03-19T03:01:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.192+0000 7f476de00640 1 -- 172.21.15.92:0/2912138064 wait complete. 2024-03-19T03:01:01.213 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-03-19T03:01:01.213 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-19T03:01:01.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:01.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.422+0000 7ff667c00640 1 Processor -- start 2024-03-19T03:01:01.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.423+0000 7ff667c00640 1 -- start start 2024-03-19T03:01:01.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.424+0000 7ff667c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff6600c6470 0x7ff6600c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:01.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.424+0000 7ff667c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6600c9a80 con 0x7ff660152bf0 2024-03-19T03:01:01.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.424+0000 7ff667c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6600c8650 con 0x7ff6601593d0 2024-03-19T03:01:01.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.424+0000 7ff667c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff6600c6c90 con 0x7ff6600c6470 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff665200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff660152bf0 0x7ff66006eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34186/0 (socket says 172.21.15.92:34186) 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff665200640 1 -- 172.21.15.92:0/2522409550 learned_addr learned my addr 172.21.15.92:0/2522409550 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65fe00640 1 --2- 172.21.15.92:0/2522409550 >> v2:172.21.15.159:3300/0 conn(0x7ff6600c6470 0x7ff6600c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570910027 0 0) 0x7ff6600c8650 con 0x7ff6601593d0 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff648003520 con 0x7ff6601593d0 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628129858 0 0) 0x7ff6600c9a80 con 0x7ff660152bf0 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65fe00640 1 -- 172.21.15.92:0/2522409550 >> v1:172.21.15.159:6789/0 conn(0x7ff6601593d0 legacy=0x7ff6600c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65fe00640 1 -- 172.21.15.92:0/2522409550 >> v1:172.21.15.92:6789/0 conn(0x7ff660152bf0 legacy=0x7ff66006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.425+0000 7ff65fe00640 1 -- 172.21.15.92:0/2522409550 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6600c8930 con 0x7ff6600c6470 2024-03-19T03:01:01.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.426+0000 7ff65fe00640 1 --2- 172.21.15.92:0/2522409550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff6600c3f80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3955d576ff0c5782 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:01.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.427+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c035020 con 0x7ff6600c6470 2024-03-19T03:01:01.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.427+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64c031830 con 0x7ff6600c6470 2024-03-19T03:01:01.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.427+0000 7ff65f400640 1 -- 172.21.15.92:0/2522409550 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c031be0 con 0x7ff6600c6470 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.427+0000 7ff667c00640 1 -- 172.21.15.92:0/2522409550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 msgr2=0x7ff6600c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.427+0000 7ff667c00640 1 --2- 172.21.15.92:0/2522409550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff6600c3f80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c02b8d0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.428+0000 7ff667c00640 1 -- 172.21.15.92:0/2522409550 shutdown_connections 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.428+0000 7ff667c00640 1 --2- 172.21.15.92:0/2522409550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff6600c3f80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.428+0000 7ff667c00640 1 -- 172.21.15.92:0/2522409550 >> 172.21.15.92:0/2522409550 conn(0x7ff6600c8d90 msgr2=0x7ff6600c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:01.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.428+0000 7ff667c00640 1 -- 172.21.15.92:0/2522409550 shutdown_connections 2024-03-19T03:01:01.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.428+0000 7ff667c00640 1 -- 172.21.15.92:0/2522409550 wait complete. 2024-03-19T03:01:01.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.429+0000 7ff667c00640 1 Processor -- start 2024-03-19T03:01:01.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.429+0000 7ff667c00640 1 -- start start 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff66015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff660152bf0 0x7ff6601e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff66006e4a0 con 0x7ff660152bf0 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff660159910 con 0x7ff6600c6470 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.430+0000 7ff667c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff66006ea60 con 0x7ff6601593d0 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff66015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff65fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff660152bf0 0x7ff6601e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:01.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:53206/0 (socket says 172.21.15.92:53206) 2024-03-19T03:01:01.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 -- 172.21.15.92:0/3057845414 learned_addr learned my addr 172.21.15.92:0/3057845414 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 -- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 msgr2=0x7ff66015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff66015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 -- 172.21.15.92:0/3057845414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff660152bf0 msgr2=0x7ff6601e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff660152bf0 0x7ff6601e1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 -- 172.21.15.92:0/3057845414 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6601522b0 con 0x7ff6601593d0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665200640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff66015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.431+0000 7ff665c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff65800c6b0 tx=0x7ff65800cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.432+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658003b70 con 0x7ff6601593d0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.432+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff658003d10 con 0x7ff6601593d0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.432+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66015a300 con 0x7ff6601593d0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.432+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658005310 con 0x7ff6601593d0 2024-03-19T03:01:01.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.432+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66015a8c0 con 0x7ff6601593d0 2024-03-19T03:01:01.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.433+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6600684c0 con 0x7ff6601593d0 2024-03-19T03:01:01.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.434+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff6580054b0 con 0x7ff6601593d0 2024-03-19T03:01:01.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.435+0000 7ff65cc00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 0x7ff64006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:01.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.435+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7ff65808aa60 con 0x7ff6601593d0 2024-03-19T03:01:01.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.435+0000 7ff665200640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 0x7ff64006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:01.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.436+0000 7ff665200640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 0x7ff64006e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff654007fd0 tx=0x7ff654007240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:01.440 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.440+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff658058cf0 con 0x7ff6601593d0 2024-03-19T03:01:01.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.678+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6600c3b50 con 0x7ff6601593d0 2024-03-19T03:01:01.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.679+0000 7ff65cc00640 1 -- 172.21.15.92:0/3057845414 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6003 (secure 0 0 0) 0x7ff65805cd90 con 0x7ff6601593d0 2024-03-19T03:01:01.680 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:01.680 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:00:59.656375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:01.680 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 21 2024-03-19T03:01:01.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 msgr2=0x7ff64006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 0x7ff64006e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff654007fd0 tx=0x7ff654007240 comp rx=0 tx=0).stop 2024-03-19T03:01:01.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 msgr2=0x7ff6601e2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:01.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff65800c6b0 tx=0x7ff65800cb70 comp rx=0 tx=0).stop 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 shutdown_connections 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff64006bbd0 0x7ff64006e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6601593d0 0x7ff6601e2530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff660152bf0 0x7ff6601e1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 --2- 172.21.15.92:0/3057845414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6600c6470 0x7ff66015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 >> 172.21.15.92:0/3057845414 conn(0x7ff6600c8d90 msgr2=0x7ff6600c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 shutdown_connections 2024-03-19T03:01:01.683 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:01.682+0000 7ff667c00640 1 -- 172.21.15.92:0/3057845414 wait complete. 2024-03-19T03:01:01.703 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5041": { "addr": "172.21.15.159:6837/2066707194", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 2066707194, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 2066707194, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5041, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:00:59.656375+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5041 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:01:03.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:03.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.919+0000 7f70af800640 1 Processor -- start 2024-03-19T03:01:03.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.919+0000 7f70af800640 1 -- start start 2024-03-19T03:01:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.920+0000 7f70af800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70a8151900 0x7f70a8158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.920+0000 7f70af800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70a80c7da0 0x7f70a80c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.920+0000 7f70af800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a80c3b80 con 0x7f70a8151ce0 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.920+0000 7f70af800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f70a8158860 con 0x7f70a8151900 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.920+0000 7f70af800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f70a8072e70 con 0x7f70a80c7da0 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70a8151900 0x7f70a8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ad800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f70a80c7da0 0x7f70a80c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f709fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f70a8151ce0 0x7f70a80723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52130/0 (socket says 172.21.15.92:52130) 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f709fe00640 1 -- 172.21.15.92:0/935500516 learned_addr learned my addr 172.21.15.92:0/935500516 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70a8151900 0x7f70a8158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33320/0 (socket says 172.21.15.92:33320) 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 -- 172.21.15.92:0/935500516 >> v2:172.21.15.159:3300/0 conn(0x7f70a80c7da0 msgr2=0x7f70a80c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 --2- 172.21.15.92:0/935500516 >> v2:172.21.15.159:3300/0 conn(0x7f70a80c7da0 0x7f70a80c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 -- 172.21.15.92:0/935500516 >> v1:172.21.15.159:6789/0 conn(0x7f70a8151ce0 legacy=0x7f70a80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70ace00640 1 -- 172.21.15.92:0/935500516 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a8073150 con 0x7f70a8151900 2024-03-19T03:01:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.921+0000 7f70a7e00640 1 -- 172.21.15.92:0/935500516 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650345533 0 0) 0x7f70a80c3b80 con 0x7f70a8151ce0 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.922+0000 7f70ace00640 1 --2- 172.21.15.92:0/935500516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151900 0x7f70a8158320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70a8073700 tx=0x7f709802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6e2277897bea7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.922+0000 7f70a7e00640 1 -- 172.21.15.92:0/935500516 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098034070 con 0x7f70a8151900 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70a7e00640 1 -- 172.21.15.92:0/935500516 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7098030610 con 0x7f70a8151900 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70a7e00640 1 -- 172.21.15.92:0/935500516 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70980309e0 con 0x7f70a8151900 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 -- 172.21.15.92:0/935500516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151900 msgr2=0x7f70a8158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 --2- 172.21.15.92:0/935500516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151900 0x7f70a8158320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70a8073700 tx=0x7f709802b960 comp rx=0 tx=0).stop 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 -- 172.21.15.92:0/935500516 shutdown_connections 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 --2- 172.21.15.92:0/935500516 >> v2:172.21.15.159:3300/0 conn(0x7f70a80c7da0 0x7f70a80c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 --2- 172.21.15.92:0/935500516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151900 0x7f70a8158320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 -- 172.21.15.92:0/935500516 >> 172.21.15.92:0/935500516 conn(0x7f70a80c8d90 msgr2=0x7f70a80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 -- 172.21.15.92:0/935500516 shutdown_connections 2024-03-19T03:01:03.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.923+0000 7f70af800640 1 -- 172.21.15.92:0/935500516 wait complete. 2024-03-19T03:01:03.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.925+0000 7f70af800640 1 Processor -- start 2024-03-19T03:01:03.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.925+0000 7f70af800640 1 -- start start 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80c7da0 0x7f70a81e0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 0x7f70a81574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70a80c9af0 con 0x7f70a8151ce0 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8071eb0 con 0x7f70a8151900 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70af800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f70a80c9c70 con 0x7f70a80c7da0 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70ad800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70ad800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33326/0 (socket says 172.21.15.92:33326) 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f709fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 0x7f70a81574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70ad800640 1 -- 172.21.15.92:0/2635661078 learned_addr learned my addr 172.21.15.92:0/2635661078 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:03.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.926+0000 7f70ace00640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80c7da0 0x7f70a81e0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 -- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80c7da0 msgr2=0x7f70a81e0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:03.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80c7da0 0x7f70a81e0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 -- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 msgr2=0x7f70a81574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 0x7f70a81574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 -- 172.21.15.92:0/2635661078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a814a710 con 0x7f70a8151ce0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f709fe00640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 0x7f70a81574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70ad800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70a000bf40 tx=0x7f70a000e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a000b2d0 con 0x7f70a8151ce0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a001b070 con 0x7f70a8151ce0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0009780 con 0x7f70a8151ce0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.927+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a8156d10 con 0x7f70a8151ce0 2024-03-19T03:01:03.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.928+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a814ae90 con 0x7f70a8151ce0 2024-03-19T03:01:03.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.929+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a80684c0 con 0x7f70a8151ce0 2024-03-19T03:01:03.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.929+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f70a0005d10 con 0x7f70a8151ce0 2024-03-19T03:01:03.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.931+0000 7f70a5600640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 0x7f708406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:03.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.931+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f70a008d250 con 0x7f70a8151ce0 2024-03-19T03:01:03.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.931+0000 7f70ace00640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 0x7f708406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:03.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.932+0000 7f70ace00640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 0x7f708406e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7098036000 tx=0x7f7098008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:03.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:03.935+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f70a0012040 con 0x7f70a8151ce0 2024-03-19T03:01:04.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.172+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70a80c4b80 con 0x7f70a8151ce0 2024-03-19T03:01:04.174 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.174+0000 7f70a5600640 1 -- 172.21.15.92:0/2635661078 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6003 (secure 0 0 0) 0x7f70a005b450 con 0x7f70a8151ce0 2024-03-19T03:01:04.174 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:04.174 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:00:59.656375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:04.175 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 21 2024-03-19T03:01:04.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 msgr2=0x7f708406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 0x7f708406e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7098036000 tx=0x7f7098008030 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 msgr2=0x7f70a814b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70a000bf40 tx=0x7f70a000e550 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 shutdown_connections 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f708406bc00 0x7f708406e0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a8151ce0 0x7f70a814b6a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70a8151900 0x7f70a81574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 --2- 172.21.15.92:0/2635661078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70a80c7da0 0x7f70a81e0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 >> 172.21.15.92:0/2635661078 conn(0x7f70a80c8d90 msgr2=0x7f70a8072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 shutdown_connections 2024-03-19T03:01:04.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:04.176+0000 7f70af800640 1 -- 172.21.15.92:0/2635661078 wait complete. 2024-03-19T03:01:04.195 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:01:06.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:06.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.416+0000 7fee8e200640 1 Processor -- start 2024-03-19T03:01:06.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.416+0000 7fee8e200640 1 -- start start 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.417+0000 7fee8e200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fee88151900 0x7fee88158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.417+0000 7fee8e200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 0x7fee880723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.417+0000 7fee8e200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee880c3b80 con 0x7fee880c7da0 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee8e200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fee88158860 con 0x7fee88151ce0 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee8e200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fee88072e70 con 0x7fee88151900 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fee880c7da0 0x7fee8815adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:45992/0 (socket says 172.21.15.92:45992) 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87e00640 1 -- 172.21.15.92:0/2155377134 learned_addr learned my addr 172.21.15.92:0/2155377134 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87400640 1 --2- 172.21.15.92:0/2155377134 >> v2:172.21.15.159:3300/0 conn(0x7fee88151900 0x7fee88158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee86a00640 1 --2- 172.21.15.92:0/2155377134 >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 0x7fee880723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee86000640 1 -- 172.21.15.92:0/2155377134 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329606677 0 0) 0x7fee880c3b80 con 0x7fee880c7da0 2024-03-19T03:01:06.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee86000640 1 -- 172.21.15.92:0/2155377134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee6c003520 con 0x7fee880c7da0 2024-03-19T03:01:06.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87400640 1 -- 172.21.15.92:0/2155377134 >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 msgr2=0x7fee880723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:06.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87400640 1 --2- 172.21.15.92:0/2155377134 >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 0x7fee880723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87400640 1 -- 172.21.15.92:0/2155377134 >> v1:172.21.15.159:6789/0 conn(0x7fee880c7da0 legacy=0x7fee8815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee87400640 1 -- 172.21.15.92:0/2155377134 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee88073150 con 0x7fee88151900 2024-03-19T03:01:06.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.418+0000 7fee86a00640 1 --2- 172.21.15.92:0/2155377134 >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 0x7fee880723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:06.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.419+0000 7fee87400640 1 --2- 172.21.15.92:0/2155377134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee88151900 0x7fee88158320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fee88075650 tx=0x7fee7802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c35e8d5bdbc350b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:06.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee86000640 1 -- 172.21.15.92:0/2155377134 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78034070 con 0x7fee88151900 2024-03-19T03:01:06.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee86000640 1 -- 172.21.15.92:0/2155377134 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee78030610 con 0x7fee88151900 2024-03-19T03:01:06.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee86000640 1 -- 172.21.15.92:0/2155377134 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee780309e0 con 0x7fee88151900 2024-03-19T03:01:06.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 -- 172.21.15.92:0/2155377134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee88151900 msgr2=0x7fee88158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 --2- 172.21.15.92:0/2155377134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee88151900 0x7fee88158320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fee88075650 tx=0x7fee7802b960 comp rx=0 tx=0).stop 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 -- 172.21.15.92:0/2155377134 shutdown_connections 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 --2- 172.21.15.92:0/2155377134 >> v2:172.21.15.92:3300/0 conn(0x7fee88151ce0 0x7fee880723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 --2- 172.21.15.92:0/2155377134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee88151900 0x7fee88158320 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.420+0000 7fee8e200640 1 -- 172.21.15.92:0/2155377134 >> 172.21.15.92:0/2155377134 conn(0x7fee880c8d90 msgr2=0x7fee880c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.421+0000 7fee8e200640 1 -- 172.21.15.92:0/2155377134 shutdown_connections 2024-03-19T03:01:06.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.421+0000 7fee8e200640 1 -- 172.21.15.92:0/2155377134 wait complete. 2024-03-19T03:01:06.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.423+0000 7fee8e200640 1 Processor -- start 2024-03-19T03:01:06.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.423+0000 7fee8e200640 1 -- start start 2024-03-19T03:01:06.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 0x7fee8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee88151ce0 0x7fee88147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fee88072910 con 0x7fee88151ce0 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fee88071eb0 con 0x7fee880c7da0 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee8e200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fee880c9af0 con 0x7fee88151900 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee87e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee88151ce0 0x7fee88147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee86a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 0x7fee8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee87400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee87400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:42362/0 (socket says 172.21.15.92:42362) 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee86a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 0x7fee8806ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36070/0 (socket says 172.21.15.92:36070) 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.424+0000 7fee87400640 1 -- 172.21.15.92:0/449689090 learned_addr learned my addr 172.21.15.92:0/449689090 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 -- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 msgr2=0x7fee8806ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 0x7fee8806ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 -- 172.21.15.92:0/449689090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee88151ce0 msgr2=0x7fee88147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee88151ce0 0x7fee88147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 -- 172.21.15.92:0/449689090 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee881ef8c0 con 0x7fee880c7da0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee87400640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fee78009860 tx=0x7fee780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78046020 con 0x7fee880c7da0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee78030410 con 0x7fee880c7da0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee881efba0 con 0x7fee880c7da0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78034040 con 0x7fee880c7da0 2024-03-19T03:01:06.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.425+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881f0160 con 0x7fee880c7da0 2024-03-19T03:01:06.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.426+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee880684c0 con 0x7fee880c7da0 2024-03-19T03:01:06.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.427+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fee78037070 con 0x7fee880c7da0 2024-03-19T03:01:06.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.428+0000 7fee7f400640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 0x7fee6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:06.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.428+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fee780ac730 con 0x7fee880c7da0 2024-03-19T03:01:06.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.428+0000 7fee86a00640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 0x7fee6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:06.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.429+0000 7fee86a00640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 0x7fee6806e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee8814f8f0 tx=0x7fee74009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:06.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.435+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fee78050020 con 0x7fee880c7da0 2024-03-19T03:01:06.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.675+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee880c5e00 con 0x7fee880c7da0 2024-03-19T03:01:06.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.676+0000 7fee7f400640 1 -- 172.21.15.92:0/449689090 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6003 (secure 0 0 0) 0x7fee780305e0 con 0x7fee880c7da0 2024-03-19T03:01:06.677 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:06.677 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:00:59.656375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:06.677 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 21 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 msgr2=0x7fee6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 0x7fee6806e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee8814f8f0 tx=0x7fee74009190 comp rx=0 tx=0).stop 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 msgr2=0x7fee88147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fee78009860 tx=0x7fee780046c0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 shutdown_connections 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fee6806bbd0 0x7fee6806e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee88151ce0 0x7fee88147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee88151900 0x7fee8806ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 --2- 172.21.15.92:0/449689090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee880c7da0 0x7fee88147f60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:06.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 >> 172.21.15.92:0/449689090 conn(0x7fee880c8d90 msgr2=0x7fee880c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:06.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.680+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 shutdown_connections 2024-03-19T03:01:06.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:06.681+0000 7fee8e200640 1 -- 172.21.15.92:0/449689090 wait complete. 2024-03-19T03:01:06.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:01:08.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:08.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.919+0000 7f2e20400640 1 Processor -- start 2024-03-19T03:01:08.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.919+0000 7f2e20400640 1 -- start start 2024-03-19T03:01:08.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e20400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 0x7f2e1806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:08.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e20400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e180c9a80 con 0x7f2e181593d0 2024-03-19T03:01:08.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e20400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e180c8650 con 0x7f2e180c6470 2024-03-19T03:01:08.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e20400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2e180c6c90 con 0x7f2e18152bf0 2024-03-19T03:01:08.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e1e400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2e181593d0 0x7f2e180c4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:37452/0 (socket says 172.21.15.92:37452) 2024-03-19T03:01:08.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e1e400640 1 -- 172.21.15.92:0/3789091849 learned_addr learned my addr 172.21.15.92:0/3789091849 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:08.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e1da00640 1 --2- 172.21.15.92:0/3789091849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 0x7f2e1806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.921+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1933303397 0 0) 0x7f2e180c9a80 con 0x7f2e181593d0 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e00003520 con 0x7f2e181593d0 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777146096 0 0) 0x7f2e180c8650 con 0x7f2e180c6470 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e180c9a80 con 0x7f2e180c6470 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e1da00640 1 -- 172.21.15.92:0/3789091849 >> v1:172.21.15.159:6789/0 conn(0x7f2e180c6470 legacy=0x7f2e180c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e1da00640 1 -- 172.21.15.92:0/3789091849 >> v1:172.21.15.92:6789/0 conn(0x7f2e181593d0 legacy=0x7f2e180c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:08.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.922+0000 7f2e1da00640 1 -- 172.21.15.92:0/3789091849 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e180c8930 con 0x7f2e18152bf0 2024-03-19T03:01:08.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.923+0000 7f2e1da00640 1 --2- 172.21.15.92:0/3789091849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 0x7f2e1806eef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0097d0 tx=0x7f2e0c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6eb55fcad0ea84 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:08.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.923+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c035020 con 0x7f2e18152bf0 2024-03-19T03:01:08.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.923+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e0c031800 con 0x7f2e18152bf0 2024-03-19T03:01:08.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.923+0000 7f2e17e00640 1 -- 172.21.15.92:0/3789091849 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c031bb0 con 0x7f2e18152bf0 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 -- 172.21.15.92:0/3789091849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 msgr2=0x7f2e1806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 --2- 172.21.15.92:0/3789091849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 0x7f2e1806eef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0097d0 tx=0x7f2e0c02b8a0 comp rx=0 tx=0).stop 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 -- 172.21.15.92:0/3789091849 shutdown_connections 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 --2- 172.21.15.92:0/3789091849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e18152bf0 0x7f2e1806eef0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 -- 172.21.15.92:0/3789091849 >> 172.21.15.92:0/3789091849 conn(0x7f2e180c8d90 msgr2=0x7f2e180c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 -- 172.21.15.92:0/3789091849 shutdown_connections 2024-03-19T03:01:08.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.924+0000 7f2e20400640 1 -- 172.21.15.92:0/3789091849 wait complete. 2024-03-19T03:01:08.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.926+0000 7f2e20400640 1 Processor -- start 2024-03-19T03:01:08.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.926+0000 7f2e20400640 1 -- start start 2024-03-19T03:01:08.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180c6470 0x7f2e18144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:08.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:08.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e181593d0 0x7f2e18156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1806e4a0 con 0x7f2e18152bf0 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18159910 con 0x7f2e180c6470 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e20400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1806ea60 con 0x7f2e181593d0 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40174/0 (socket says 172.21.15.92:40174) 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 -- 172.21.15.92:0/2841024930 learned_addr learned my addr 172.21.15.92:0/2841024930 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1e400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e181593d0 0x7f2e18156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1da00640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180c6470 0x7f2e18144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 -- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e181593d0 msgr2=0x7f2e18156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e181593d0 0x7f2e18156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 -- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180c6470 msgr2=0x7f2e18144c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:08.930 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180c6470 0x7f2e18144c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.927+0000 7f2e1d000640 1 -- 172.21.15.92:0/2841024930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e18157ad0 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e1d000640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400c500 tx=0x7f2e0400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e040052d0 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e04005490 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04012870 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e181577f0 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.928+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e181f11f0 con 0x7f2e18152bf0 2024-03-19T03:01:08.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.929+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e180684c0 con 0x7f2e18152bf0 2024-03-19T03:01:08.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.930+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e04012a10 con 0x7f2e18152bf0 2024-03-19T03:01:08.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.931+0000 7f2e15600640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 0x7f2df806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:08.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.931+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f2e0408a760 con 0x7f2e18152bf0 2024-03-19T03:01:08.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.931+0000 7f2e1da00640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 0x7f2df806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:08.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.932+0000 7f2e1da00640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 0x7f2df806e0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c036000 tx=0x7f2e0c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:08.937 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:08.936+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e04058a80 con 0x7f2e18152bf0 2024-03-19T03:01:09.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.175+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e18151f60 con 0x7f2e18152bf0 2024-03-19T03:01:09.177 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.175+0000 7f2e15600640 1 -- 172.21.15.92:0/2841024930 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6007 (secure 0 0 0) 0x7f2e0405cb20 con 0x7f2e18152bf0 2024-03-19T03:01:09.177 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:09.177 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:07.488849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:09.177 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 22 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 msgr2=0x7f2df806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 0x7f2df806e0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c036000 tx=0x7f2e0c008030 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 msgr2=0x7f2e18158180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400c500 tx=0x7f2e0400c9c0 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 shutdown_connections 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2df806bc00 0x7f2df806e0c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e181593d0 0x7f2e18156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e18152bf0 0x7f2e18158180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 --2- 172.21.15.92:0/2841024930 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e180c6470 0x7f2e18144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 >> 172.21.15.92:0/2841024930 conn(0x7f2e180c8d90 msgr2=0x7f2e181515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 shutdown_connections 2024-03-19T03:01:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:09.178+0000 7f2e20400640 1 -- 172.21.15.92:0/2841024930 wait complete. 2024-03-19T03:01:09.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:01:11.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:11.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.418+0000 7fd065e00640 1 Processor -- start 2024-03-19T03:01:11.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.418+0000 7fd065e00640 1 -- start start 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd065e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd060152bf0 0x7fd06006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd065e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0600c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd065e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0600c9a80 con 0x7fd0600c6470 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd065e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd0600c6c90 con 0x7fd060152bf0 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd065e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd060159910 con 0x7fd0601593d0 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd056000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd0600c6470 0x7fd0600c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:37466/0 (socket says 172.21.15.92:37466) 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd056000640 1 -- 172.21.15.92:0/1260034974 learned_addr learned my addr 172.21.15.92:0/1260034974 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:11.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd05ea00640 1 --2- 172.21.15.92:0/1260034974 >> v2:172.21.15.92:3300/0 conn(0x7fd060152bf0 0x7fd06006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.420+0000 7fd05f400640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0600c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05e000640 1 -- 172.21.15.92:0/1260034974 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460283505 0 0) 0x7fd0600c9a80 con 0x7fd0600c6470 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05e000640 1 -- 172.21.15.92:0/1260034974 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd044003520 con 0x7fd0600c6470 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05ea00640 1 -- 172.21.15.92:0/1260034974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 msgr2=0x7fd0600c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05ea00640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0600c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05ea00640 1 -- 172.21.15.92:0/1260034974 >> v1:172.21.15.92:6789/0 conn(0x7fd0600c6470 legacy=0x7fd0600c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.421+0000 7fd05ea00640 1 -- 172.21.15.92:0/1260034974 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600c8650 con 0x7fd060152bf0 2024-03-19T03:01:11.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.422+0000 7fd05ea00640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd06006eef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0500097a0 tx=0x7fd05002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8c08e6670c4dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:11.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.422+0000 7fd05e000640 1 -- 172.21.15.92:0/1260034974 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500315a0 con 0x7fd060152bf0 2024-03-19T03:01:11.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.422+0000 7fd05e000640 1 -- 172.21.15.92:0/1260034974 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd050037070 con 0x7fd060152bf0 2024-03-19T03:01:11.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.422+0000 7fd05e000640 1 -- 172.21.15.92:0/1260034974 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050031940 con 0x7fd060152bf0 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 -- 172.21.15.92:0/1260034974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 msgr2=0x7fd06006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd06006eef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0500097a0 tx=0x7fd05002b840 comp rx=0 tx=0).stop 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 -- 172.21.15.92:0/1260034974 shutdown_connections 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0600c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 --2- 172.21.15.92:0/1260034974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd06006eef0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 -- 172.21.15.92:0/1260034974 >> 172.21.15.92:0/1260034974 conn(0x7fd0600c8d90 msgr2=0x7fd0600c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 -- 172.21.15.92:0/1260034974 shutdown_connections 2024-03-19T03:01:11.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.423+0000 7fd065e00640 1 -- 172.21.15.92:0/1260034974 wait complete. 2024-03-19T03:01:11.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.425+0000 7fd065e00640 1 Processor -- start 2024-03-19T03:01:11.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.425+0000 7fd065e00640 1 -- start start 2024-03-19T03:01:11.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600c6470 0x7fd060072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0601f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd06006ea60 con 0x7fd060152bf0 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd06006e4a0 con 0x7fd0600c6470 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd065e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd06006e6f0 con 0x7fd0601593d0 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40196/0 (socket says 172.21.15.92:40196) 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 -- 172.21.15.92:0/161404876 learned_addr learned my addr 172.21.15.92:0/161404876 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd05f400640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0601f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd05ea00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600c6470 0x7fd060072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 -- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 msgr2=0x7fd0601f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0601f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 -- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600c6470 msgr2=0x7fd060072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600c6470 0x7fd060072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.426+0000 7fd056000640 1 -- 172.21.15.92:0/161404876 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0601f0d20 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd056000640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c460 tx=0x7fd04800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048017070 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd048005ce0 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0480072f0 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0601f1000 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.427+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0601f15c0 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.428+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0600684c0 con 0x7fd060152bf0 2024-03-19T03:01:11.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.429+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd04801a070 con 0x7fd060152bf0 2024-03-19T03:01:11.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.430+0000 7fd057e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 0x7fd03c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:11.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.430+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fd048089fe0 con 0x7fd060152bf0 2024-03-19T03:01:11.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.431+0000 7fd05ea00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 0x7fd03c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:11.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.431+0000 7fd05ea00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 0x7fd03c06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd050036000 tx=0x7fd05003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:11.436 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.435+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd048058290 con 0x7fd060152bf0 2024-03-19T03:01:11.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.673+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd06006f950 con 0x7fd060152bf0 2024-03-19T03:01:11.676 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.675+0000 7fd057e00640 1 -- 172.21.15.92:0/161404876 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6007 (secure 0 0 0) 0x7fd04805c330 con 0x7fd060152bf0 2024-03-19T03:01:11.676 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:11.677 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:07.488849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:11.677 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 22 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.677+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 msgr2=0x7fd03c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.677+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 0x7fd03c06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd050036000 tx=0x7fd05003d040 comp rx=0 tx=0).stop 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.677+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 msgr2=0x7fd0600cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.677+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c460 tx=0x7fd04800c920 comp rx=0 tx=0).stop 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 shutdown_connections 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd03c06bbd0 0x7fd03c06e090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0601593d0 0x7fd0601f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd060152bf0 0x7fd0600cb400 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 --2- 172.21.15.92:0/161404876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0600c6470 0x7fd060072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:11.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 >> 172.21.15.92:0/161404876 conn(0x7fd0600c8d90 msgr2=0x7fd06014dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:11.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 shutdown_connections 2024-03-19T03:01:11.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:11.678+0000 7fd065e00640 1 -- 172.21.15.92:0/161404876 wait complete. 2024-03-19T03:01:11.696 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:01:13.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:13.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.911+0000 7f3252400640 1 Processor -- start 2024-03-19T03:01:13.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.911+0000 7f3252400640 1 -- start start 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.912+0000 7f3252400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 0x7f324c0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.912+0000 7f3252400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f324c1593d0 0x7f324c0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.912+0000 7f3252400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f324c0c9a80 con 0x7f324c152bf0 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.912+0000 7f3252400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f324c0c6c90 con 0x7f324c1593d0 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.912+0000 7f3252400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f324c159910 con 0x7f324c0c6470 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f3242a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 0x7f324c0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324b400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f324c152bf0 0x7f324c06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:46010/0 (socket says 172.21.15.92:46010) 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324b400640 1 -- 172.21.15.92:0/4016974156 learned_addr learned my addr 172.21.15.92:0/4016974156 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324be00640 1 --2- 172.21.15.92:0/4016974156 >> v2:172.21.15.92:3300/0 conn(0x7f324c1593d0 0x7f324c0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324aa00640 1 -- 172.21.15.92:0/4016974156 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547088577 0 0) 0x7f324c0c9a80 con 0x7f324c152bf0 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324aa00640 1 -- 172.21.15.92:0/4016974156 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3230003520 con 0x7f324c152bf0 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324be00640 1 -- 172.21.15.92:0/4016974156 >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 msgr2=0x7f324c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324be00640 1 --2- 172.21.15.92:0/4016974156 >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 0x7f324c0c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324be00640 1 -- 172.21.15.92:0/4016974156 >> v1:172.21.15.159:6789/0 conn(0x7f324c152bf0 legacy=0x7f324c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f324be00640 1 -- 172.21.15.92:0/4016974156 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c0c8650 con 0x7f324c1593d0 2024-03-19T03:01:13.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.913+0000 7f3242a00640 1 --2- 172.21.15.92:0/4016974156 >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 0x7f324c0c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:13.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.914+0000 7f324be00640 1 --2- 172.21.15.92:0/4016974156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c1593d0 0x7f324c0c44c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f324c075340 tx=0x7f324402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d7e0184613bc21 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:13.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f324aa00640 1 -- 172.21.15.92:0/4016974156 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244039070 con 0x7f324c1593d0 2024-03-19T03:01:13.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f324aa00640 1 -- 172.21.15.92:0/4016974156 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f324402cbf0 con 0x7f324c1593d0 2024-03-19T03:01:13.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f324aa00640 1 -- 172.21.15.92:0/4016974156 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244034440 con 0x7f324c1593d0 2024-03-19T03:01:13.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 -- 172.21.15.92:0/4016974156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c1593d0 msgr2=0x7f324c0c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 --2- 172.21.15.92:0/4016974156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c1593d0 0x7f324c0c44c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f324c075340 tx=0x7f324402be40 comp rx=0 tx=0).stop 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 -- 172.21.15.92:0/4016974156 shutdown_connections 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 --2- 172.21.15.92:0/4016974156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c1593d0 0x7f324c0c44c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 --2- 172.21.15.92:0/4016974156 >> v2:172.21.15.159:3300/0 conn(0x7f324c0c6470 0x7f324c0c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.915+0000 7f3252400640 1 -- 172.21.15.92:0/4016974156 >> 172.21.15.92:0/4016974156 conn(0x7f324c0c8d90 msgr2=0x7f324c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.916+0000 7f3252400640 1 -- 172.21.15.92:0/4016974156 shutdown_connections 2024-03-19T03:01:13.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.916+0000 7f3252400640 1 -- 172.21.15.92:0/4016974156 wait complete. 2024-03-19T03:01:13.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.917+0000 7f3252400640 1 Processor -- start 2024-03-19T03:01:13.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.917+0000 7f3252400640 1 -- start start 2024-03-19T03:01:13.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f324c152bf0 0x7f324c1eedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 0x7f324c1ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f324c06ea60 con 0x7f324c0c6470 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f324c06e4a0 con 0x7f324c152bf0 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.918+0000 7f3252400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f324c06e6f0 con 0x7f324c1593d0 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 0x7f324c1ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40220/0 (socket says 172.21.15.92:40220) 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324b400640 1 -- 172.21.15.92:0/977090528 learned_addr learned my addr 172.21.15.92:0/977090528 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f3242a00640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f324c152bf0 0x7f324c1eedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 -- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f324c152bf0 msgr2=0x7f324c1eedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f324c152bf0 0x7f324c1eedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:13.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 -- 172.21.15.92:0/977090528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 msgr2=0x7f324c1e5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 -- 172.21.15.92:0/977090528 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c1f2b50 con 0x7f324c1593d0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324b400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.919+0000 7f324be00640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 0x7f324c1ef300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f324c0c9c50 tx=0x7f3244004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.920+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244039050 con 0x7f324c1593d0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.920+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3244045070 con 0x7f324c1593d0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.920+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244009450 con 0x7f324c1593d0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.920+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c1f1fd0 con 0x7f324c1593d0 2024-03-19T03:01:13.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.920+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c1f2590 con 0x7f324c1593d0 2024-03-19T03:01:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.921+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324c0684c0 con 0x7f324c1593d0 2024-03-19T03:01:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.922+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f324403e850 con 0x7f324c1593d0 2024-03-19T03:01:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.923+0000 7f3243e00640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 0x7f322806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.923+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f32440ad870 con 0x7f324c1593d0 2024-03-19T03:01:13.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.923+0000 7f324b400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 0x7f322806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:13.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.924+0000 7f324b400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 0x7f322806e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f323c005880 tx=0x7f323c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:13.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:13.927+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f324407bc20 con 0x7f324c1593d0 2024-03-19T03:01:14.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.164+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f324c1ee700 con 0x7f324c1593d0 2024-03-19T03:01:14.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.165+0000 7f3243e00640 1 -- 172.21.15.92:0/977090528 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6004 (secure 0 0 0) 0x7f324407fcc0 con 0x7f324c1593d0 2024-03-19T03:01:14.167 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:14.167 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:12.693903+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:14.167 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 23 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 msgr2=0x7f322806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 0x7f322806e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f323c005880 tx=0x7f323c009040 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 msgr2=0x7f324c1ef300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 0x7f324c1ef300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f324c0c9c50 tx=0x7f3244004a40 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 shutdown_connections 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f322806bbd0 0x7f322806e090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f324c1593d0 0x7f324c1ef300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f324c152bf0 0x7f324c1eedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 --2- 172.21.15.92:0/977090528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f324c0c6470 0x7f324c1e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:14.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 >> 172.21.15.92:0/977090528 conn(0x7f324c0c8d90 msgr2=0x7f324c0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:14.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 shutdown_connections 2024-03-19T03:01:14.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:14.168+0000 7f3252400640 1 -- 172.21.15.92:0/977090528 wait complete. 2024-03-19T03:01:14.190 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5041": { "addr": "172.21.15.159:6837/2066707194", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 2066707194, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 2066707194, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5041, "incarnation": 21, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:01:12.693903+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5041 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:01:16.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:16.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.408+0000 7ffb2d400640 1 Processor -- start 2024-03-19T03:01:16.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.408+0000 7ffb2d400640 1 -- start start 2024-03-19T03:01:16.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb28072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 0x7ffb281599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.411 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ffb28159520 con 0x7ffb280c36f0 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ffb28072140 con 0x7ffb280c5860 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.409+0000 7ffb2d400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffb280c4ef0 con 0x7ffb280c5480 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb2ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb28072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb1f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 0x7ffb281599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb2ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33478/0 (socket says 172.21.15.92:33478) 2024-03-19T03:01:16.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb2ca00640 1 -- 172.21.15.92:0/2944369191 learned_addr learned my addr 172.21.15.92:0/2944369191 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 -- 172.21.15.92:0/2944369191 >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 msgr2=0x7ffb281599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 0x7ffb281599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 -- 172.21.15.92:0/2944369191 >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 msgr2=0x7ffb28159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb27e00640 1 -- 172.21.15.92:0/2944369191 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb280728d0 con 0x7ffb280c5480 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb1f400640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 0x7ffb281599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.410+0000 7ffb2ca00640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:16.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.411+0000 7ffb27e00640 1 --2- 172.21.15.92:0/2944369191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb28072390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffb2806fb40 tx=0x7ffb1802b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69bb846b8530626 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:16.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.412+0000 7ffb27400640 1 -- 172.21.15.92:0/2944369191 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18038070 con 0x7ffb280c5480 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.412+0000 7ffb27400640 1 -- 172.21.15.92:0/2944369191 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb18031520 con 0x7ffb280c5480 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.412+0000 7ffb27400640 1 -- 172.21.15.92:0/2944369191 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb180318f0 con 0x7ffb280c5480 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.412+0000 7ffb2d400640 1 -- 172.21.15.92:0/2944369191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 msgr2=0x7ffb28072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.412+0000 7ffb2d400640 1 --2- 172.21.15.92:0/2944369191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb28072390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffb2806fb40 tx=0x7ffb1802b7c0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 -- 172.21.15.92:0/2944369191 shutdown_connections 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.92:3300/0 conn(0x7ffb280c36f0 0x7ffb28159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 --2- 172.21.15.92:0/2944369191 >> v2:172.21.15.159:3300/0 conn(0x7ffb280c5860 0x7ffb281599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 --2- 172.21.15.92:0/2944369191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb28072390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 -- 172.21.15.92:0/2944369191 >> 172.21.15.92:0/2944369191 conn(0x7ffb280cc1f0 msgr2=0x7ffb280cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:16.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 -- 172.21.15.92:0/2944369191 shutdown_connections 2024-03-19T03:01:16.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.413+0000 7ffb2d400640 1 -- 172.21.15.92:0/2944369191 wait complete. 2024-03-19T03:01:16.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.415+0000 7ffb2d400640 1 Processor -- start 2024-03-19T03:01:16.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.415+0000 7ffb2d400640 1 -- start start 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb281ea0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb280c5860 0x7ffb28143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffb28158ed0 con 0x7ffb280c36f0 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffb28071e60 con 0x7ffb280c5860 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2d400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffb281591b0 con 0x7ffb280c5480 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb27e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb27e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33484/0 (socket says 172.21.15.92:33484) 2024-03-19T03:01:16.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb27e00640 1 -- 172.21.15.92:0/4208785259 learned_addr learned my addr 172.21.15.92:0/4208785259 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb1f400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb281ea0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.416+0000 7ffb2ca00640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb280c5860 0x7ffb28143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 -- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 msgr2=0x7ffb281ea0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb281ea0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 -- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb280c5860 msgr2=0x7ffb28143dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb280c5860 0x7ffb28143dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 -- 172.21.15.92:0/4208785259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb28150eb0 con 0x7ffb280c36f0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb27e00640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009890 tx=0x7ffb180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18042020 con 0x7ffb280c36f0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb1803d890 con 0x7ffb280c36f0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb281ece60 con 0x7ffb280c36f0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.417+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18038040 con 0x7ffb280c36f0 2024-03-19T03:01:16.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.418+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb28143300 con 0x7ffb280c36f0 2024-03-19T03:01:16.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.419+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb2804f580 con 0x7ffb280c36f0 2024-03-19T03:01:16.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.419+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffb18045020 con 0x7ffb280c36f0 2024-03-19T03:01:16.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.420+0000 7ffb24c00640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:16.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.420+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7ffb180ac3c0 con 0x7ffb280c36f0 2024-03-19T03:01:16.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.421+0000 7ffb1f400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:16.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.421+0000 7ffb1f400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 0x7ffb0406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009560 tx=0x7ffb10009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:16.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.425+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ffb1804c030 con 0x7ffb280c36f0 2024-03-19T03:01:16.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.664+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb280c3bd0 con 0x7ffb280c36f0 2024-03-19T03:01:16.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.665+0000 7ffb24c00640 1 -- 172.21.15.92:0/4208785259 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6004 (secure 0 0 0) 0x7ffb1807a650 con 0x7ffb280c36f0 2024-03-19T03:01:16.667 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:16.667 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:12.693903+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:01:16.667 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 23 2024-03-19T03:01:16.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 msgr2=0x7ffb0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 0x7ffb0406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009560 tx=0x7ffb10009190 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 msgr2=0x7ffb28145860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009890 tx=0x7ffb180046c0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 shutdown_connections 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb280c5860 0x7ffb28143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffb280c5480 0x7ffb281ea0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 --2- 172.21.15.92:0/4208785259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb280c36f0 0x7ffb28145860 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 >> 172.21.15.92:0/4208785259 conn(0x7ffb280cc1f0 msgr2=0x7ffb281471d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 shutdown_connections 2024-03-19T03:01:16.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:16.668+0000 7ffb2d400640 1 -- 172.21.15.92:0/4208785259 wait complete. 2024-03-19T03:01:16.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:01:18.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:18.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.907+0000 7fa132a00640 1 Processor -- start 2024-03-19T03:01:18.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.907+0000 7fa132a00640 1 -- start start 2024-03-19T03:01:18.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa132a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa12c0c7ca0 0x7fa12c06f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa132a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa12c06fcd0 0x7fa12c0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa132a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa12c06bf50 con 0x7fa12c158550 2024-03-19T03:01:18.911 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa132a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa12c158930 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa132a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fa12c0cd260 con 0x7fa12c06fcd0 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa12c0c7ca0 0x7fa12c06f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa12c0c7ca0 0x7fa12c06f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33492/0 (socket says 172.21.15.92:33492) 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 -- 172.21.15.92:0/2372475857 learned_addr learned my addr 172.21.15.92:0/2372475857 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12be00640 1 --1- 172.21.15.92:0/2372475857 >> v1:172.21.15.159:6789/0 conn(0x7fa12c158550 0x7fa12c06b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:35968/0 (socket says 172.21.15.92:35968) 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa130a00640 1 --2- 172.21.15.92:0/2372475857 >> v2:172.21.15.159:3300/0 conn(0x7fa12c06fcd0 0x7fa12c0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 -- 172.21.15.92:0/2372475857 >> v2:172.21.15.159:3300/0 conn(0x7fa12c06fcd0 msgr2=0x7fa12c0c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 --2- 172.21.15.92:0/2372475857 >> v2:172.21.15.159:3300/0 conn(0x7fa12c06fcd0 0x7fa12c0c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:18.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 -- 172.21.15.92:0/2372475857 >> v1:172.21.15.159:6789/0 conn(0x7fa12c158550 legacy=0x7fa12c06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:18.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12aa00640 1 -- 172.21.15.92:0/2372475857 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42153640 0 0) 0x7fa12c06bf50 con 0x7fa12c158550 2024-03-19T03:01:18.913 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.909+0000 7fa12b400640 1 -- 172.21.15.92:0/2372475857 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c070630 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa12b400640 1 --2- 172.21.15.92:0/2372475857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c0c7ca0 0x7fa12c06f790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0cb1a0 tx=0x7fa11802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201e4920482d277e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa12aa00640 1 -- 172.21.15.92:0/2372475857 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11802f850 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa12aa00640 1 -- 172.21.15.92:0/2372475857 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa11802f9f0 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa12aa00640 1 -- 172.21.15.92:0/2372475857 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11802fdc0 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 -- 172.21.15.92:0/2372475857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c0c7ca0 msgr2=0x7fa12c06f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 --2- 172.21.15.92:0/2372475857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c0c7ca0 0x7fa12c06f790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0cb1a0 tx=0x7fa11802ba30 comp rx=0 tx=0).stop 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 -- 172.21.15.92:0/2372475857 shutdown_connections 2024-03-19T03:01:18.914 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 --2- 172.21.15.92:0/2372475857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c0c7ca0 0x7fa12c06f790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:18.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 --2- 172.21.15.92:0/2372475857 >> v2:172.21.15.159:3300/0 conn(0x7fa12c06fcd0 0x7fa12c0c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:18.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.911+0000 7fa132a00640 1 -- 172.21.15.92:0/2372475857 >> 172.21.15.92:0/2372475857 conn(0x7fa12c072100 msgr2=0x7fa12c0724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:18.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.912+0000 7fa132a00640 1 -- 172.21.15.92:0/2372475857 shutdown_connections 2024-03-19T03:01:18.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.912+0000 7fa132a00640 1 -- 172.21.15.92:0/2372475857 wait complete. 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.913+0000 7fa132a00640 1 Processor -- start 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.913+0000 7fa132a00640 1 -- start start 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12c158550 0x7fa12c1f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c0cd7e0 con 0x7fa12c06fcd0 2024-03-19T03:01:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c0cd500 con 0x7fa12c158550 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.914+0000 7fa132a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa12c0cdac0 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa130a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12c158550 0x7fa12c1f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33504/0 (socket says 172.21.15.92:33504) 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:46960/0 (socket says 172.21.15.92:46960) 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12be00640 1 -- 172.21.15.92:0/1789468618 learned_addr learned my addr 172.21.15.92:0/1789468618 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 -- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12c158550 msgr2=0x7fa12c1f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12c158550 0x7fa12c1f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 -- 172.21.15.92:0/1789468618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 msgr2=0x7fa12c0c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12b400640 1 -- 172.21.15.92:0/1789468618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c152800 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.915+0000 7fa12be00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:18.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa12b400640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa118033000 tx=0x7fa118004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:18.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118037bc0 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa118046070 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118041400 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c152c80 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.916+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c1f0340 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.917+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12c0768a0 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.918+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa118040070 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.919+0000 7fa123e00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 0x7fa10c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.919+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fa1180ad140 con 0x7fa12c0c7ca0 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.919+0000 7fa12be00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 0x7fa10c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.920+0000 7fa12be00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 0x7fa10c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa11c000f60 tx=0x7fa11c007e10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:18.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:18.924+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa118033030 con 0x7fa12c0c7ca0 2024-03-19T03:01:19.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.154+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa12c06f570 con 0x7fa12c0c7ca0 2024-03-19T03:01:19.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.155+0000 7fa123e00640 1 -- 172.21.15.92:0/1789468618 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6011 (secure 0 0 0) 0x7fa11807b3d0 con 0x7fa12c0c7ca0 2024-03-19T03:01:19.157 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:19.157 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:01:19.157 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 24 2024-03-19T03:01:19.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 msgr2=0x7fa10c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:19.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 0x7fa10c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa11c000f60 tx=0x7fa11c007e10 comp rx=0 tx=0).stop 2024-03-19T03:01:19.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 msgr2=0x7fa12c1522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa118033000 tx=0x7fa118004730 comp rx=0 tx=0).stop 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 shutdown_connections 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa10c06bbd0 0x7fa10c06e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12c158550 0x7fa12c1f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa12c0c7ca0 0x7fa12c1522c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 --2- 172.21.15.92:0/1789468618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa12c06fcd0 0x7fa12c0c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 >> 172.21.15.92:0/1789468618 conn(0x7fa12c072100 msgr2=0x7fa12c0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 shutdown_connections 2024-03-19T03:01:19.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:19.158+0000 7fa132a00640 1 -- 172.21.15.92:0/1789468618 wait complete. 2024-03-19T03:01:19.178 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4688 2024-03-19T03:01:19.178 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.d 2024-03-19T03:01:23.184 INFO:tasks.ceph.mds.f.smithi159.stderr:2024-03-19T03:01:23.183+0000 7f70e2f83640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T03:01:23.962 INFO:tasks.workunit.client.0.smithi092.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-03-19T03:01:23.963 INFO:tasks.workunit.client.0.smithi092.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-03-19T03:01:23.963 INFO:tasks.workunit.client.0.smithi092.stderr:fatal: early EOF 2024-03-19T03:01:23.964 INFO:tasks.workunit.client.0.smithi092.stderr:fatal: fetch-pack: invalid index-pack output 2024-03-19T03:01:24.087 INFO:tasks.workunit.client.0.smithi092.stderr:+ retry 2024-03-19T03:01:24.087 INFO:tasks.workunit.client.0.smithi092.stderr:+ rm -rf ceph 2024-03-19T03:01:24.091 INFO:tasks.workunit.client.0.smithi092.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-03-19T03:01:24.099 INFO:tasks.workunit.client.0.smithi092.stderr:Cloning into 'ceph'... 2024-03-19T03:01:27.179 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-03-19T03:01:27.179 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-19T03:01:27.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-19T03:01:27.182 INFO:tasks.ceph.mds.d:Started 2024-03-19T03:01:27.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:27.259 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-19T03:01:27.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.388+0000 7f60b8800640 1 Processor -- start 2024-03-19T03:01:27.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.388+0000 7f60b8800640 1 -- start start 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.389+0000 7f60b8800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f60b015b550 0x7f60b00cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.389+0000 7f60b8800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b015b9e0 con 0x7f60b0154b10 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.389+0000 7f60b8800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b00c5340 con 0x7f60b00c63e0 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.389+0000 7f60b8800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f60b00c5620 con 0x7f60b015b550 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b5400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f60b015b550 0x7f60b00cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b6800640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f60b00c63e0 0x7f60b006c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:34386/0 (socket says 172.21.15.92:34386) 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b5e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f60b0154b10 0x7f60b015c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49004/0 (socket says 172.21.15.92:49004) 2024-03-19T03:01:27.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b6800640 1 -- 172.21.15.92:0/2952773570 learned_addr learned my addr 172.21.15.92:0/2952773570 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:01:27.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b5400640 1 -- 172.21.15.92:0/2952773570 >> v1:172.21.15.159:6789/0 conn(0x7f60b00c63e0 legacy=0x7f60b006c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b5400640 1 -- 172.21.15.92:0/2952773570 >> v1:172.21.15.92:6789/0 conn(0x7f60b0154b10 legacy=0x7f60b015c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b5400640 1 -- 172.21.15.92:0/2952773570 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b00c5900 con 0x7f60b015b550 2024-03-19T03:01:27.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.390+0000 7f60b4a00640 1 -- 172.21.15.92:0/2952773570 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793309156 0 0) 0x7f60b00c5340 con 0x7f60b00c63e0 2024-03-19T03:01:27.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.391+0000 7f60b5400640 1 --2- 172.21.15.92:0/2952773570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b015b550 0x7f60b00cae40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097a0 tx=0x7f60a002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a6797b144921af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:27.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b4a00640 1 -- 172.21.15.92:0/2952773570 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a002f810 con 0x7f60b015b550 2024-03-19T03:01:27.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b4a00640 1 -- 172.21.15.92:0/2952773570 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a002f9b0 con 0x7f60b015b550 2024-03-19T03:01:27.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b4a00640 1 -- 172.21.15.92:0/2952773570 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a002fd80 con 0x7f60b015b550 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 -- 172.21.15.92:0/2952773570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b015b550 msgr2=0x7f60b00cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 --2- 172.21.15.92:0/2952773570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b015b550 0x7f60b00cae40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097a0 tx=0x7f60a002b9f0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 -- 172.21.15.92:0/2952773570 shutdown_connections 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 --2- 172.21.15.92:0/2952773570 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b015b550 0x7f60b00cae40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 -- 172.21.15.92:0/2952773570 >> 172.21.15.92:0/2952773570 conn(0x7f60b0072970 msgr2=0x7f60b0066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 -- 172.21.15.92:0/2952773570 shutdown_connections 2024-03-19T03:01:27.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.392+0000 7f60b8800640 1 -- 172.21.15.92:0/2952773570 wait complete. 2024-03-19T03:01:27.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.394+0000 7f60b8800640 1 Processor -- start 2024-03-19T03:01:27.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.394+0000 7f60b8800640 1 -- start start 2024-03-19T03:01:27.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60b0154b10 0x7f60b00cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 0x7f60b00cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f60b0067ca0 con 0x7f60b0154b10 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00c5ec0 con 0x7f60b00cd990 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b8800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f60b0154ef0 con 0x7f60b00c63e0 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b5e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b5e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:56626/0 (socket says 172.21.15.92:56626) 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b5e00640 1 -- 172.21.15.92:0/2566665089 learned_addr learned my addr 172.21.15.92:0/2566665089 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b5400640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60b0154b10 0x7f60b00cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.395+0000 7f60b6800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 0x7f60b00cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 -- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 msgr2=0x7f60b00cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 0x7f60b00cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 -- 172.21.15.92:0/2566665089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60b0154b10 msgr2=0x7f60b00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60b0154b10 0x7f60b00cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 -- 172.21.15.92:0/2566665089 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b00c7800 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b6800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 0x7f60b00cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b5e00640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60a400c4d0 tx=0x7f60a400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4018070 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a4007480 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40049e0 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b00cbd20 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.396+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b01531c0 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.397+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b00684c0 con 0x7f60b00c63e0 2024-03-19T03:01:27.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.398+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f60a4004b80 con 0x7f60b00c63e0 2024-03-19T03:01:27.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.399+0000 7f60aa000640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 0x7f609406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:27.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.399+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f60a4089db0 con 0x7f60b00c63e0 2024-03-19T03:01:27.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.399+0000 7f60b5400640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 0x7f609406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:27.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.400+0000 7f60b5400640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 0x7f609406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f60a0036000 tx=0x7f60a0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:27.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.404+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f60a408e070 con 0x7f60b00c63e0 2024-03-19T03:01:27.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.644+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60b015d3a0 con 0x7f60b00c63e0 2024-03-19T03:01:27.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.645+0000 7f60aa000640 1 -- 172.21.15.92:0/2566665089 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6011 (secure 0 0 0) 0x7f60a4007670 con 0x7f60b00c63e0 2024-03-19T03:01:27.646 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:27.646 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:01:27.647 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 24 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 msgr2=0x7f609406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 0x7f609406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f60a0036000 tx=0x7f60a0008030 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 msgr2=0x7f60b00cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60a400c4d0 tx=0x7f60a400c990 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 shutdown_connections 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f609406bbd0 0x7f609406e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60b00cd990 0x7f60b00cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60b0154b10 0x7f60b00cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 --2- 172.21.15.92:0/2566665089 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60b00c63e0 0x7f60b00cd450 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 >> 172.21.15.92:0/2566665089 conn(0x7f60b0072970 msgr2=0x7f60b015c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 shutdown_connections 2024-03-19T03:01:27.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:27.648+0000 7f60b8800640 1 -- 172.21.15.92:0/2566665089 wait complete. 2024-03-19T03:01:27.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-19T03:01:29.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:29.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.891+0000 7fc9d2600640 1 Processor -- start 2024-03-19T03:01:29.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.891+0000 7fc9d2600640 1 -- start start 2024-03-19T03:01:29.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.892+0000 7fc9d2600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.892+0000 7fc9d2600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc9cc1593d0 0x7fc9cc0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.892+0000 7fc9d2600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9cc0c9a80 con 0x7fc9cc0c6470 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.892+0000 7fc9d2600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fc9cc0c6c90 con 0x7fc9cc1593d0 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.892+0000 7fc9d2600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc9cc159910 con 0x7fc9cc152bf0 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:56628/0 (socket says 172.21.15.92:56628) 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 -- 172.21.15.92:0/2055153650 learned_addr learned my addr 172.21.15.92:0/2055153650 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cbe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc9cc1593d0 0x7fc9cc0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 -- 172.21.15.92:0/2055153650 >> v2:172.21.15.159:3300/0 conn(0x7fc9cc1593d0 msgr2=0x7fc9cc0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 --2- 172.21.15.92:0/2055153650 >> v2:172.21.15.159:3300/0 conn(0x7fc9cc1593d0 0x7fc9cc0c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 -- 172.21.15.92:0/2055153650 >> v1:172.21.15.92:6789/0 conn(0x7fc9cc0c6470 legacy=0x7fc9cc0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:29.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.893+0000 7fc9cb400640 1 -- 172.21.15.92:0/2055153650 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc0c8650 con 0x7fc9cc152bf0 2024-03-19T03:01:29.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.894+0000 7fc9cb400640 1 --2- 172.21.15.92:0/2055153650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009e50 tx=0x7fc9bc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3970fceb6c165ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:29.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.894+0000 7fc9caa00640 1 -- 172.21.15.92:0/2055153650 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc02ca70 con 0x7fc9cc152bf0 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.894+0000 7fc9caa00640 1 -- 172.21.15.92:0/2055153650 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9bc02cc10 con 0x7fc9cc152bf0 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9caa00640 1 -- 172.21.15.92:0/2055153650 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0343f0 con 0x7fc9cc152bf0 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 -- 172.21.15.92:0/2055153650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 msgr2=0x7fc9cc06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 --2- 172.21.15.92:0/2055153650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009e50 tx=0x7fc9bc02bed0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 -- 172.21.15.92:0/2055153650 shutdown_connections 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 --2- 172.21.15.92:0/2055153650 >> v2:172.21.15.159:3300/0 conn(0x7fc9cc1593d0 0x7fc9cc0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 --2- 172.21.15.92:0/2055153650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc152bf0 0x7fc9cc06eef0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 -- 172.21.15.92:0/2055153650 >> 172.21.15.92:0/2055153650 conn(0x7fc9cc0c8d90 msgr2=0x7fc9cc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 -- 172.21.15.92:0/2055153650 shutdown_connections 2024-03-19T03:01:29.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.895+0000 7fc9d2600640 1 -- 172.21.15.92:0/2055153650 wait complete. 2024-03-19T03:01:29.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.897+0000 7fc9d2600640 1 Processor -- start 2024-03-19T03:01:29.898 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.897+0000 7fc9d2600640 1 -- start start 2024-03-19T03:01:29.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.899 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 0x7fc9cc15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc06ea60 con 0x7fc9cc1593d0 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc06e4a0 con 0x7fc9cc152bf0 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9d2600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc9cc06e6f0 con 0x7fc9cc0c6470 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9c2a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 0x7fc9cc15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:56640/0 (socket says 172.21.15.92:56640) 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 -- 172.21.15.92:0/1292210886 learned_addr learned my addr 172.21.15.92:0/1292210886 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:01:29.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9c2a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:57018/0 (socket says 172.21.15.92:57018) 2024-03-19T03:01:29.901 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 -- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 msgr2=0x7fc9cc15c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 -- 172.21.15.92:0/1292210886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 msgr2=0x7fc9cc15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.898+0000 7fc9cb400640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 0x7fc9cc15ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9cb400640 1 -- 172.21.15.92:0/1292210886 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc071f90 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9c2a00640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9cbe00640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 0x7fc9cc15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9cb400640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc043020 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9bc03e450 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc009510 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9cc072270 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.899+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9cc15a440 con 0x7fc9cc0c6470 2024-03-19T03:01:29.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.900+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9cc0684c0 con 0x7fc9cc0c6470 2024-03-19T03:01:29.903 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.901+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc9bc046020 con 0x7fc9cc0c6470 2024-03-19T03:01:29.903 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.902+0000 7fc9c3e00640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 0x7fc9a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:29.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.902+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fc9bc0ad280 con 0x7fc9cc0c6470 2024-03-19T03:01:29.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.902+0000 7fc9c2a00640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 0x7fc9a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:29.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.903+0000 7fc9c2a00640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 0x7fc9a806e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc072910 tx=0x7fc9b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:29.912 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:29.911+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9bc04d030 con 0x7fc9cc0c6470 2024-03-19T03:01:30.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.140+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9cc06f950 con 0x7fc9cc0c6470 2024-03-19T03:01:30.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.142+0000 7fc9c3e00640 1 -- 172.21.15.92:0/1292210886 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7fc9bc07b510 con 0x7fc9cc0c6470 2024-03-19T03:01:30.143 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:01:30.144 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:01:30.144 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 msgr2=0x7fc9a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 0x7fc9a806e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc072910 tx=0x7fc9b4009040 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 msgr2=0x7fc9cc0cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc004a40 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 shutdown_connections 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc9a806bbd0 0x7fc9a806e090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9cc1593d0 0x7fc9cc15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9cc152bf0 0x7fc9cc15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 --2- 172.21.15.92:0/1292210886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9cc0c6470 0x7fc9cc0cb530 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 >> 172.21.15.92:0/1292210886 conn(0x7fc9cc0c8d90 msgr2=0x7fc9cc150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:30.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 shutdown_connections 2024-03-19T03:01:30.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:01:30.144+0000 7fc9d2600640 1 -- 172.21.15.92:0/1292210886 wait complete. 2024-03-19T03:01:30.162 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-03-19T03:01:30.162 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2024-03-19T03:01:30.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:01:30.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:02:00.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:02:00.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:02:31.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:02:31.316 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:03:01.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:03:01.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:03:06.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:06.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.383+0000 7f5c49000640 1 Processor -- start 2024-03-19T03:03:06.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.383+0000 7f5c49000640 1 -- start start 2024-03-19T03:03:06.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c49000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5c440c6470 0x7f5c440c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c49000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 0x7f5c440c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c49000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c440c9a80 con 0x7f5c44152bf0 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c49000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f5c440c6c90 con 0x7f5c441593d0 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c49000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5c44159910 con 0x7f5c440c6470 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c41e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5c44152bf0 0x7f5c4406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56232/0 (socket says 172.21.15.92:56232) 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c41e00640 1 -- 172.21.15.92:0/3852586290 learned_addr learned my addr 172.21.15.92:0/3852586290 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c42800640 1 --2- 172.21.15.92:0/3852586290 >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 0x7f5c440c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c41400640 1 --2- 172.21.15.92:0/3852586290 >> v2:172.21.15.159:3300/0 conn(0x7f5c440c6470 0x7f5c440c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.388 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.385+0000 7f5c40a00640 1 -- 172.21.15.92:0/3852586290 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539919140 0 0) 0x7f5c440c9a80 con 0x7f5c44152bf0 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c40a00640 1 -- 172.21.15.92:0/3852586290 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c44152bf0 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c41400640 1 -- 172.21.15.92:0/3852586290 >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 msgr2=0x7f5c440c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c41400640 1 --2- 172.21.15.92:0/3852586290 >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 0x7f5c440c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c41400640 1 -- 172.21.15.92:0/3852586290 >> v1:172.21.15.92:6789/0 conn(0x7f5c44152bf0 legacy=0x7f5c4406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c41400640 1 -- 172.21.15.92:0/3852586290 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c440c8650 con 0x7f5c440c6470 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.386+0000 7f5c42800640 1 --2- 172.21.15.92:0/3852586290 >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 0x7f5c440c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.387+0000 7f5c41400640 1 --2- 172.21.15.92:0/3852586290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c440c3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c44075340 tx=0x7f5c2c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2633f423a2374a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.387+0000 7f5c40a00640 1 -- 172.21.15.92:0/3852586290 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0315a0 con 0x7f5c440c6470 2024-03-19T03:03:06.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.387+0000 7f5c40a00640 1 -- 172.21.15.92:0/3852586290 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2c037070 con 0x7f5c440c6470 2024-03-19T03:03:06.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.387+0000 7f5c40a00640 1 -- 172.21.15.92:0/3852586290 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c031940 con 0x7f5c440c6470 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 -- 172.21.15.92:0/3852586290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 msgr2=0x7f5c440c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 --2- 172.21.15.92:0/3852586290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c440c3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c44075340 tx=0x7f5c2c02b840 comp rx=0 tx=0).stop 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 -- 172.21.15.92:0/3852586290 shutdown_connections 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 --2- 172.21.15.92:0/3852586290 >> v2:172.21.15.92:3300/0 conn(0x7f5c441593d0 0x7f5c440c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 --2- 172.21.15.92:0/3852586290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c440c3f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 -- 172.21.15.92:0/3852586290 >> 172.21.15.92:0/3852586290 conn(0x7f5c440c8d90 msgr2=0x7f5c440c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 -- 172.21.15.92:0/3852586290 shutdown_connections 2024-03-19T03:03:06.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.388+0000 7f5c49000640 1 -- 172.21.15.92:0/3852586290 wait complete. 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.390+0000 7f5c49000640 1 Processor -- start 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.390+0000 7f5c49000640 1 -- start start 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c4414e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 0x7f5c4413f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4406ea60 con 0x7f5c441593d0 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4406e4a0 con 0x7f5c440c6470 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c49000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4406e6f0 con 0x7f5c44152bf0 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c41400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c41e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c4414e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c42800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 0x7f5c4413f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c41400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:45662/0 (socket says 172.21.15.92:45662) 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c41400640 1 -- 172.21.15.92:0/1444830770 learned_addr learned my addr 172.21.15.92:0/1444830770 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c41e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c4414e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51786/0 (socket says 172.21.15.92:51786) 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.391+0000 7f5c42800640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 0x7f5c4413f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37748/0 (socket says 172.21.15.92:37748) 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 -- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 msgr2=0x7f5c4414e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c4414e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 -- 172.21.15.92:0/1444830770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 msgr2=0x7f5c4413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 0x7f5c4413f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 -- 172.21.15.92:0/1444830770 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c44151610 con 0x7f5c44152bf0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c41400640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009890 tx=0x7f5c2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c032630 con 0x7f5c44152bf0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2c0327d0 con 0x7f5c44152bf0 2024-03-19T03:03:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c032b80 con 0x7f5c44152bf0 2024-03-19T03:03:06.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c441ee870 con 0x7f5c44152bf0 2024-03-19T03:03:06.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.392+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c441eee30 con 0x7f5c44152bf0 2024-03-19T03:03:06.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.394+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c440684c0 con 0x7f5c44152bf0 2024-03-19T03:03:06.397 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.394+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5c2c04c020 con 0x7f5c44152bf0 2024-03-19T03:03:06.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.395+0000 7f5c3a000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 0x7f5c2006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.395+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f5c2c02f020 con 0x7f5c44152bf0 2024-03-19T03:03:06.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.395+0000 7f5c41e00640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 0x7f5c2006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.396+0000 7f5c41e00640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 0x7f5c2006e0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c34005880 tx=0x7f5c34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.401 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.399+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5c2c040070 con 0x7f5c44152bf0 2024-03-19T03:03:06.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.640+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c440c3b50 con 0x7f5c44152bf0 2024-03-19T03:03:06.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.641+0000 7f5c3a000640 1 -- 172.21.15.92:0/1444830770 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7f5c2c07af80 con 0x7f5c44152bf0 2024-03-19T03:03:06.643 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:06.643 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:06.644 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:06.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.644+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 msgr2=0x7f5c2006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.644+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 0x7f5c2006e0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c34005880 tx=0x7f5c34009040 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.644+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 msgr2=0x7f5c4413ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.644+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009890 tx=0x7f5c2c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 shutdown_connections 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c2006bc00 0x7f5c2006e0c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c441593d0 0x7f5c4413f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c44152bf0 0x7f5c4413ed20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 --2- 172.21.15.92:0/1444830770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c440c6470 0x7f5c4414e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 >> 172.21.15.92:0/1444830770 conn(0x7f5c440c8d90 msgr2=0x7f5c441581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 shutdown_connections 2024-03-19T03:03:06.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.645+0000 7f5c49000640 1 -- 172.21.15.92:0/1444830770 wait complete. 2024-03-19T03:03:06.665 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-19T03:03:06.665 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-19T03:03:06.665 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:03:06.677 INFO:tasks.ceph.mds.f:Stopped 2024-03-19T03:03:06.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:06.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.882+0000 7f314bc00640 1 Processor -- start 2024-03-19T03:03:06.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.882+0000 7f314bc00640 1 -- start start 2024-03-19T03:03:06.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.883+0000 7f314bc00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f31440c6470 0x7f31440c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.883+0000 7f314bc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f31441593d0 0x7f31440c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.883+0000 7f314bc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31440c9a80 con 0x7f3144152bf0 2024-03-19T03:03:06.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.883+0000 7f314bc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f31440c6c90 con 0x7f31441593d0 2024-03-19T03:03:06.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.883+0000 7f314bc00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f3144159910 con 0x7f31440c6470 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3144152bf0 0x7f314406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56234/0 (socket says 172.21.15.92:56234) 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149200640 1 -- 172.21.15.92:0/1835477171 learned_addr learned my addr 172.21.15.92:0/1835477171 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149c00640 1 --2- 172.21.15.92:0/1835477171 >> v2:172.21.15.92:3300/0 conn(0x7f31441593d0 0x7f31440c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f313be00640 1 --2- 172.21.15.92:0/1835477171 >> v2:172.21.15.159:3300/0 conn(0x7f31440c6470 0x7f31440c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149c00640 1 -- 172.21.15.92:0/1835477171 >> v2:172.21.15.159:3300/0 conn(0x7f31440c6470 msgr2=0x7f31440c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149c00640 1 --2- 172.21.15.92:0/1835477171 >> v2:172.21.15.159:3300/0 conn(0x7f31440c6470 0x7f31440c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149c00640 1 -- 172.21.15.92:0/1835477171 >> v1:172.21.15.92:6789/0 conn(0x7f3144152bf0 legacy=0x7f314406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3143e00640 1 -- 172.21.15.92:0/1835477171 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253160005 0 0) 0x7f31440c9a80 con 0x7f3144152bf0 2024-03-19T03:03:06.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.884+0000 7f3149c00640 1 -- 172.21.15.92:0/1835477171 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31440c8650 con 0x7f31441593d0 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.885+0000 7f3149c00640 1 --2- 172.21.15.92:0/1835477171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31441593d0 0x7f31440c44c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3144075340 tx=0x7f313c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9f24a66bc33035 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f3143e00640 1 -- 172.21.15.92:0/1835477171 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c039070 con 0x7f31441593d0 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f3143e00640 1 -- 172.21.15.92:0/1835477171 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f313c02cbf0 con 0x7f31441593d0 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f3143e00640 1 -- 172.21.15.92:0/1835477171 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c034440 con 0x7f31441593d0 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 -- 172.21.15.92:0/1835477171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31441593d0 msgr2=0x7f31440c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 --2- 172.21.15.92:0/1835477171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31441593d0 0x7f31440c44c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3144075340 tx=0x7f313c02be40 comp rx=0 tx=0).stop 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 -- 172.21.15.92:0/1835477171 shutdown_connections 2024-03-19T03:03:06.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 --2- 172.21.15.92:0/1835477171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31441593d0 0x7f31440c44c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 --2- 172.21.15.92:0/1835477171 >> v2:172.21.15.159:3300/0 conn(0x7f31440c6470 0x7f31440c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 -- 172.21.15.92:0/1835477171 >> 172.21.15.92:0/1835477171 conn(0x7f31440c8d90 msgr2=0x7f31440c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 -- 172.21.15.92:0/1835477171 shutdown_connections 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.886+0000 7f314bc00640 1 -- 172.21.15.92:0/1835477171 wait complete. 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.888+0000 7f314bc00640 1 Processor -- start 2024-03-19T03:03:06.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.888+0000 7f314bc00640 1 -- start start 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31440c6470 0x7f314415bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31441593d0 0x7f3144141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f314406ea60 con 0x7f3144152bf0 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f314406e4a0 con 0x7f31440c6470 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f314bc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f314406e6f0 con 0x7f31441593d0 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f313be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f313be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37776/0 (socket says 172.21.15.92:37776) 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f313be00640 1 -- 172.21.15.92:0/416625755 learned_addr learned my addr 172.21.15.92:0/416625755 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f3149c00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31441593d0 0x7f3144141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.889+0000 7f3149200640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31440c6470 0x7f314415bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 -- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31441593d0 msgr2=0x7f3144141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31441593d0 0x7f3144141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 -- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31440c6470 msgr2=0x7f314415bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31440c6470 0x7f314415bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 -- 172.21.15.92:0/416625755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314415cb40 con 0x7f3144152bf0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f313be00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e9b0 tx=0x7f312c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c017070 con 0x7f3144152bf0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c00c9a0 con 0x7f3144152bf0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314415a6c0 con 0x7f3144152bf0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c00cd90 con 0x7f3144152bf0 2024-03-19T03:03:06.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.890+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314415ac80 con 0x7f3144152bf0 2024-03-19T03:03:06.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.891+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31440684c0 con 0x7f3144152bf0 2024-03-19T03:03:06.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.892+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f312c01a070 con 0x7f3144152bf0 2024-03-19T03:03:06.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.893+0000 7f3141600640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 0x7f312006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:06.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.893+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f312c08aba0 con 0x7f3144152bf0 2024-03-19T03:03:06.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.893+0000 7f3149200640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 0x7f312006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:06.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.894+0000 7f3149200640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 0x7f312006e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f60 tx=0x7f3134007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:06.900 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:06.898+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f312c058e30 con 0x7f3144152bf0 2024-03-19T03:03:07.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.143+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31440c3b50 con 0x7f3144152bf0 2024-03-19T03:03:07.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.144+0000 7f3141600640 1 -- 172.21.15.92:0/416625755 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7f312c05ced0 con 0x7f3144152bf0 2024-03-19T03:03:07.146 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:07.146 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:07.146 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 msgr2=0x7f312006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 0x7f312006e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f60 tx=0x7f3134007d50 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 msgr2=0x7f3144151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e9b0 tx=0x7f312c00ee70 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 shutdown_connections 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f312006bbd0 0x7f312006e090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f31441593d0 0x7f3144141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144152bf0 0x7f3144151380 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 --2- 172.21.15.92:0/416625755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31440c6470 0x7f314415bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 >> 172.21.15.92:0/416625755 conn(0x7f31440c8d90 msgr2=0x7f31440c76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 shutdown_connections 2024-03-19T03:03:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:07.147+0000 7f314bc00640 1 -- 172.21.15.92:0/416625755 wait complete. 2024-03-19T03:03:07.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:09.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:09.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.400+0000 7fe24c800640 1 Processor -- start 2024-03-19T03:03:09.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.400+0000 7fe24c800640 1 -- start start 2024-03-19T03:03:09.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.402+0000 7fe24c800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe244071da0 0x7fe2440c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.402+0000 7fe24c800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.402+0000 7fe24c800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe244072890 con 0x7fe24406f800 2024-03-19T03:03:09.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.402+0000 7fe24c800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fe244159780 con 0x7fe244071da0 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.402+0000 7fe24c800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe2440c6c90 con 0x7fe244158db0 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.403+0000 7fe249e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe244071da0 0x7fe2440c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.403+0000 7fe24a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.403+0000 7fe24a800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:45684/0 (socket says 172.21.15.92:45684) 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.403+0000 7fe24a800640 1 -- 172.21.15.92:0/2600618280 learned_addr learned my addr 172.21.15.92:0/2600618280 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.403+0000 7fe249e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe244071da0 0x7fe2440c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51822/0 (socket says 172.21.15.92:51822) 2024-03-19T03:03:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.404+0000 7fe249e00640 1 -- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 msgr2=0x7fe244159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.404+0000 7fe249e00640 1 --2- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.404+0000 7fe249e00640 1 -- 172.21.15.92:0/2600618280 >> v1:172.21.15.92:6789/0 conn(0x7fe24406f800 legacy=0x7fe244072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.404+0000 7fe249e00640 1 -- 172.21.15.92:0/2600618280 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2440c8200 con 0x7fe244071da0 2024-03-19T03:03:09.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.406+0000 7fe249e00640 1 --2- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe244071da0 0x7fe2440c8740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797faa2572fa2436 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:09.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.407+0000 7fe248a00640 1 -- 172.21.15.92:0/2600618280 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238035020 con 0x7fe244071da0 2024-03-19T03:03:09.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.407+0000 7fe248a00640 1 -- 172.21.15.92:0/2600618280 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe238031810 con 0x7fe244071da0 2024-03-19T03:03:09.410 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.407+0000 7fe248a00640 1 -- 172.21.15.92:0/2600618280 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238031be0 con 0x7fe244071da0 2024-03-19T03:03:09.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.409+0000 7fe24c800640 1 -- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe244071da0 msgr2=0x7fe2440c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.409+0000 7fe24c800640 1 --2- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe244071da0 0x7fe2440c8740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b8b0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.413+0000 7fe24c800640 1 -- 172.21.15.92:0/2600618280 shutdown_connections 2024-03-19T03:03:09.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.413+0000 7fe24c800640 1 --2- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.413+0000 7fe24c800640 1 --2- 172.21.15.92:0/2600618280 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe244071da0 0x7fe2440c8740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.413+0000 7fe24c800640 1 -- 172.21.15.92:0/2600618280 >> 172.21.15.92:0/2600618280 conn(0x7fe2440c8d90 msgr2=0x7fe2440c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:09.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.414+0000 7fe24c800640 1 -- 172.21.15.92:0/2600618280 shutdown_connections 2024-03-19T03:03:09.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.415+0000 7fe24c800640 1 -- 172.21.15.92:0/2600618280 wait complete. 2024-03-19T03:03:09.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.418+0000 7fe24c800640 1 Processor -- start 2024-03-19T03:03:09.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.418+0000 7fe24c800640 1 -- start start 2024-03-19T03:03:09.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe244071da0 0x7fe2440c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe24415fde0 con 0x7fe244071da0 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe24415fb00 con 0x7fe24406f800 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24c800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe2440c7120 con 0x7fe244158db0 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe249e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe249e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51830/0 (socket says 172.21.15.92:51830) 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe249e00640 1 -- 172.21.15.92:0/164187755 learned_addr learned my addr 172.21.15.92:0/164187755 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe24a800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.420+0000 7fe249400640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe244071da0 0x7fe2440c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 -- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 msgr2=0x7fe2440c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 -- 172.21.15.92:0/164187755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe244071da0 msgr2=0x7fe2440c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe244071da0 0x7fe2440c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 -- 172.21.15.92:0/164187755 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2441563a0 con 0x7fe244158db0 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe249e00640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe24a800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244155a80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe24000eba0 tx=0x7fe24000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.421+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24000cb20 con 0x7fe244158db0 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.422+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24413a3c0 con 0x7fe244158db0 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.422+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe24000ccc0 con 0x7fe244158db0 2024-03-19T03:03:09.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.422+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240003bb0 con 0x7fe244158db0 2024-03-19T03:03:09.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.422+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24413a940 con 0x7fe244158db0 2024-03-19T03:03:09.425 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.424+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2440684c0 con 0x7fe244158db0 2024-03-19T03:03:09.426 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.425+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe240012050 con 0x7fe244158db0 2024-03-19T03:03:09.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.425+0000 7fe23e000640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 0x7fe22806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:09.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.426+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fe24008b010 con 0x7fe244158db0 2024-03-19T03:03:09.427 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.426+0000 7fe249e00640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 0x7fe22806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:09.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.427+0000 7fe249e00640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 0x7fe22806e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe23802b8f0 tx=0x7fe238008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:09.432 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.431+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe240010040 con 0x7fe244158db0 2024-03-19T03:03:09.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.671+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2440c9310 con 0x7fe244158db0 2024-03-19T03:03:09.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.672+0000 7fe23e000640 1 -- 172.21.15.92:0/164187755 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7fe240059290 con 0x7fe244158db0 2024-03-19T03:03:09.674 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:09.674 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:09.675 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:09.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 msgr2=0x7fe22806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 0x7fe22806e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe23802b8f0 tx=0x7fe238008030 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 msgr2=0x7fe244155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244155a80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe24000eba0 tx=0x7fe24000c490 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 shutdown_connections 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fe22806bbd0 0x7fe22806e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe244158db0 0x7fe244155a80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe244071da0 0x7fe2440c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.675+0000 7fe24c800640 1 --2- 172.21.15.92:0/164187755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe24406f800 0x7fe2440c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.676+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 >> 172.21.15.92:0/164187755 conn(0x7fe2440c8d90 msgr2=0x7fe2440c7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.676+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 shutdown_connections 2024-03-19T03:03:09.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:09.676+0000 7fe24c800640 1 -- 172.21.15.92:0/164187755 wait complete. 2024-03-19T03:03:09.696 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:11.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:11.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.913+0000 7f0d17e00640 1 Processor -- start 2024-03-19T03:03:11.915 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.913+0000 7f0d17e00640 1 -- start start 2024-03-19T03:03:11.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.914+0000 7f0d17e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0d181549c0 0x7f0d180cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:11.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d17e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1815ac30 con 0x7f0d180b3180 2024-03-19T03:03:11.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d17e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1815e280 con 0x7f0d180cc300 2024-03-19T03:03:11.916 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d17e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0d1815e030 con 0x7f0d181549c0 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d16000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0d180b3180 0x7f0d180cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56252/0 (socket says 172.21.15.92:56252) 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d16000640 1 -- 172.21.15.92:0/1156993367 learned_addr learned my addr 172.21.15.92:0/1156993367 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.915+0000 7f0d16a00640 1 --2- 172.21.15.92:0/1156993367 >> v2:172.21.15.92:3300/0 conn(0x7f0d181549c0 0x7f0d180cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.916+0000 7f0d16a00640 1 -- 172.21.15.92:0/1156993367 >> v1:172.21.15.159:6789/0 conn(0x7f0d180cc300 legacy=0x7f0d181653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.916+0000 7f0d16a00640 1 -- 172.21.15.92:0/1156993367 >> v1:172.21.15.92:6789/0 conn(0x7f0d180b3180 legacy=0x7f0d180cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:11.917 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.916+0000 7f0d15600640 1 -- 172.21.15.92:0/1156993367 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740297368 0 0) 0x7f0d1815e280 con 0x7f0d180cc300 2024-03-19T03:03:11.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.916+0000 7f0d16a00640 1 -- 172.21.15.92:0/1156993367 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1815e560 con 0x7f0d181549c0 2024-03-19T03:03:11.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d16a00640 1 --2- 172.21.15.92:0/1156993367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d180cccf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0d080097a0 tx=0x7f0d0802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af92042af859f6b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:11.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d15600640 1 -- 172.21.15.92:0/1156993367 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08037070 con 0x7f0d181549c0 2024-03-19T03:03:11.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d15600640 1 -- 172.21.15.92:0/1156993367 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d08030710 con 0x7f0d181549c0 2024-03-19T03:03:11.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d15600640 1 -- 172.21.15.92:0/1156993367 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08030ae0 con 0x7f0d181549c0 2024-03-19T03:03:11.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d17e00640 1 -- 172.21.15.92:0/1156993367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 msgr2=0x7f0d180cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:11.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.917+0000 7f0d17e00640 1 --2- 172.21.15.92:0/1156993367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d180cccf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0d080097a0 tx=0x7f0d0802b710 comp rx=0 tx=0).stop 2024-03-19T03:03:11.919 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.918+0000 7f0d17e00640 1 -- 172.21.15.92:0/1156993367 shutdown_connections 2024-03-19T03:03:11.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.918+0000 7f0d17e00640 1 --2- 172.21.15.92:0/1156993367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d180cccf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:11.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.918+0000 7f0d17e00640 1 -- 172.21.15.92:0/1156993367 >> 172.21.15.92:0/1156993367 conn(0x7f0d18072970 msgr2=0x7f0d18066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:11.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.918+0000 7f0d17e00640 1 -- 172.21.15.92:0/1156993367 shutdown_connections 2024-03-19T03:03:11.920 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.918+0000 7f0d17e00640 1 -- 172.21.15.92:0/1156993367 wait complete. 2024-03-19T03:03:11.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.919+0000 7f0d17e00640 1 Processor -- start 2024-03-19T03:03:11.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 -- start start 2024-03-19T03:03:11.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d180b3180 0x7f0d18152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:11.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:11.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d18146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:11.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1815ee00 con 0x7f0d181549c0 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1815eb20 con 0x7f0d180cc300 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.920+0000 7f0d17e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d18154da0 con 0x7f0d180b3180 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d18146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d180b3180 0x7f0d18152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:51844/0 (socket says 172.21.15.92:51844) 2024-03-19T03:03:11.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 -- 172.21.15.92:0/2808966156 learned_addr learned my addr 172.21.15.92:0/2808966156 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:11.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 -- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d180b3180 msgr2=0x7f0d18152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:11.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d180b3180 0x7f0d18152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 -- 172.21.15.92:0/2808966156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 msgr2=0x7f0d18146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d18146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d16000640 1 -- 172.21.15.92:0/2808966156 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1815d6a0 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.921+0000 7f0d17400640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d18146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d16000640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c780 tx=0x7f0d0400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04017070 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d04005a90 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04007070 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1815cd30 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.922+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d18153020 con 0x7f0d180cc300 2024-03-19T03:03:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.923+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d181f04b0 con 0x7f0d180cc300 2024-03-19T03:03:11.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.924+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0d0401a070 con 0x7f0d180cc300 2024-03-19T03:03:11.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.925+0000 7f0d0ea00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 0x7f0cf806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:11.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.925+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f0d0408a150 con 0x7f0d180cc300 2024-03-19T03:03:11.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.925+0000 7f0d16a00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 0x7f0cf806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:11.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.926+0000 7f0d16a00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 0x7f0cf806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d08034000 tx=0x7f0d08008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:11.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:11.930+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0d181f04b0 con 0x7f0d180cc300 2024-03-19T03:03:12.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.170+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d1804f580 con 0x7f0d180cc300 2024-03-19T03:03:12.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.171+0000 7f0d0ea00640 1 -- 172.21.15.92:0/2808966156 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7f0d040583d0 con 0x7f0d180cc300 2024-03-19T03:03:12.173 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:12.173 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:12.173 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:12.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.173+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 msgr2=0x7f0cf806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:12.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 0x7f0cf806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d08034000 tx=0x7f0d08008030 comp rx=0 tx=0).stop 2024-03-19T03:03:12.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 msgr2=0x7f0d181f0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:12.175 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c780 tx=0x7f0d0400cc40 comp rx=0 tx=0).stop 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 shutdown_connections 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0cf806bbd0 0x7f0cf806e090 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d181549c0 0x7f0d18146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d180cc300 0x7f0d181f0b70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 --2- 172.21.15.92:0/2808966156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d180b3180 0x7f0d18152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 >> 172.21.15.92:0/2808966156 conn(0x7f0d18072970 msgr2=0x7f0d180cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 shutdown_connections 2024-03-19T03:03:12.176 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:12.174+0000 7f0d17e00640 1 -- 172.21.15.92:0/2808966156 wait complete. 2024-03-19T03:03:12.195 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:14.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:14.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.410+0000 7f182d200640 1 Processor -- start 2024-03-19T03:03:14.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.410+0000 7f182d200640 1 -- start start 2024-03-19T03:03:14.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.411+0000 7f182d200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18280c5480 0x7f1828072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:14.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.411+0000 7f182d200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1828072140 con 0x7f18280c5860 2024-03-19T03:03:14.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.411+0000 7f182d200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18280c4ef0 con 0x7f182815a0c0 2024-03-19T03:03:14.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.411+0000 7f182d200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f1828159520 con 0x7f18280c5480 2024-03-19T03:03:14.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1827400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18280c5480 0x7f1828072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:14.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f181ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f18280c5860 0x7f18281599b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:56268/0 (socket says 172.21.15.92:56268) 2024-03-19T03:03:14.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1827e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f182815a0c0 0x7f1828151610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:42368/0 (socket says 172.21.15.92:42368) 2024-03-19T03:03:14.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f181ea00640 1 -- 172.21.15.92:0/3589222685 learned_addr learned my addr 172.21.15.92:0/3589222685 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 711421521 0 0) 0x7f1820001110 con 0x7f182815a0c0 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f180c003520 con 0x7f182815a0c0 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859571321 0 0) 0x7f1828072140 con 0x7f18280c5860 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1820001110 con 0x7f18280c5860 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1827400640 1 -- 172.21.15.92:0/3589222685 >> v1:172.21.15.159:6789/0 conn(0x7f182815a0c0 legacy=0x7f1828151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1827400640 1 -- 172.21.15.92:0/3589222685 >> v1:172.21.15.92:6789/0 conn(0x7f18280c5860 legacy=0x7f18281599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.412+0000 7f1827400640 1 -- 172.21.15.92:0/3589222685 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18280728d0 con 0x7f18280c5480 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.413+0000 7f1827400640 1 --2- 172.21.15.92:0/3589222685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f1828072390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1818009d90 tx=0x7f181802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46e24f21cbea537d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818039070 con 0x7f18280c5480 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181802cc60 con 0x7f18280c5480 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f1826a00640 1 -- 172.21.15.92:0/3589222685 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818034660 con 0x7f18280c5480 2024-03-19T03:03:14.416 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 -- 172.21.15.92:0/3589222685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 msgr2=0x7f1828072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 --2- 172.21.15.92:0/3589222685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f1828072390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1818009d90 tx=0x7f181802beb0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 -- 172.21.15.92:0/3589222685 shutdown_connections 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 --2- 172.21.15.92:0/3589222685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f1828072390 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 -- 172.21.15.92:0/3589222685 >> 172.21.15.92:0/3589222685 conn(0x7f18280cc1f0 msgr2=0x7f18280cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 -- 172.21.15.92:0/3589222685 shutdown_connections 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.414+0000 7f182d200640 1 -- 172.21.15.92:0/3589222685 wait complete. 2024-03-19T03:03:14.417 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.416+0000 7f182d200640 1 Processor -- start 2024-03-19T03:03:14.418 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.416+0000 7f182d200640 1 -- start start 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f182813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18280c5860 0x7f18281ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1828158ed0 con 0x7f182815a0c0 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1828071e60 con 0x7f18280c5480 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.417+0000 7f182d200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18281591b0 con 0x7f18280c5860 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:14.419 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f182813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:14.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f181ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18280c5860 0x7f18281ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:14.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37806/0 (socket says 172.21.15.92:37806) 2024-03-19T03:03:14.420 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 -- 172.21.15.92:0/3759541097 learned_addr learned my addr 172.21.15.92:0/3759541097 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 -- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18280c5860 msgr2=0x7f18281ee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18280c5860 0x7f18281ee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 -- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 msgr2=0x7f182813ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f182813ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 -- 172.21.15.92:0/3759541097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18281eeae0 con 0x7f182815a0c0 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827400640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f182813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f1827e00640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f182000c700 tx=0x7f182000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.418+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820018070 con 0x7f182815a0c0 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.419+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1820007270 con 0x7f182815a0c0 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.419+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18281eedc0 con 0x7f182815a0c0 2024-03-19T03:03:14.421 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.419+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18200047e0 con 0x7f182815a0c0 2024-03-19T03:03:14.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.419+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182814fa70 con 0x7f182815a0c0 2024-03-19T03:03:14.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.420+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182804f580 con 0x7f182815a0c0 2024-03-19T03:03:14.422 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.421+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f182001b070 con 0x7f182815a0c0 2024-03-19T03:03:14.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.422+0000 7f181fe00640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 0x7f180406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:14.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.422+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f1820089e70 con 0x7f182815a0c0 2024-03-19T03:03:14.423 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.422+0000 7f1827400640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 0x7f180406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:14.424 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.423+0000 7f1827400640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 0x7f180406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f1818030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:14.428 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.427+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f18280c4ef0 con 0x7f182815a0c0 2024-03-19T03:03:14.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.754+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f182814b6c0 con 0x7f182815a0c0 2024-03-19T03:03:14.756 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.755+0000 7f181fe00640 1 -- 172.21.15.92:0/3759541097 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7f18200582d0 con 0x7f182815a0c0 2024-03-19T03:03:14.756 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:14.757 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:14.757 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:14.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.757+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 msgr2=0x7f180406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.757+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 0x7f180406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f1818030000 comp rx=0 tx=0).stop 2024-03-19T03:03:14.759 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.757+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 msgr2=0x7f182814c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.757+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f182000c700 tx=0x7f182000cbc0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 shutdown_connections 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f180406bbd0 0x7f180406e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f182815a0c0 0x7f182814c480 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18280c5860 0x7f18281ee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 --2- 172.21.15.92:0/3759541097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18280c5480 0x7f182813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 >> 172.21.15.92:0/3759541097 conn(0x7f18280cc1f0 msgr2=0x7f182815ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 shutdown_connections 2024-03-19T03:03:14.760 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:14.758+0000 7f182d200640 1 -- 172.21.15.92:0/3759541097 wait complete. 2024-03-19T03:03:14.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:16.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.997+0000 7fa919600640 1 Processor -- start 2024-03-19T03:03:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.997+0000 7fa919600640 1 -- start start 2024-03-19T03:03:17.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa919600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa914072380 0x7fa914071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa919600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa9140c60d0 0x7fa9140c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.000 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa919600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa91415b740 con 0x7fa914154dc0 2024-03-19T03:03:17.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa919600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa91415ba20 con 0x7fa914072380 2024-03-19T03:03:17.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa919600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fa9140c8400 con 0x7fa9140c60d0 2024-03-19T03:03:17.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa912000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa914072380 0x7fa914071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:17.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa912a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa914154dc0 0x7fa914071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:47998/0 (socket says 172.21.15.92:47998) 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa912a00640 1 -- 172.21.15.92:0/2559992885 learned_addr learned my addr 172.21.15.92:0/2559992885 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:16.999+0000 7fa913400640 1 --2- 172.21.15.92:0/2559992885 >> v2:172.21.15.159:3300/0 conn(0x7fa9140c60d0 0x7fa9140c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.000+0000 7fa913400640 1 -- 172.21.15.92:0/2559992885 >> v2:172.21.15.92:3300/0 conn(0x7fa914072380 msgr2=0x7fa914071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.000+0000 7fa913400640 1 --2- 172.21.15.92:0/2559992885 >> v2:172.21.15.92:3300/0 conn(0x7fa914072380 0x7fa914071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.000+0000 7fa913400640 1 -- 172.21.15.92:0/2559992885 >> v1:172.21.15.92:6789/0 conn(0x7fa914154dc0 legacy=0x7fa914071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.000+0000 7fa911600640 1 -- 172.21.15.92:0/2559992885 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666397215 0 0) 0x7fa91415b740 con 0x7fa914154dc0 2024-03-19T03:03:17.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.000+0000 7fa913400640 1 -- 172.21.15.92:0/2559992885 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9140c86e0 con 0x7fa9140c60d0 2024-03-19T03:03:17.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.001+0000 7fa913400640 1 --2- 172.21.15.92:0/2559992885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9140c60d0 0x7fa9140c6c90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa914075340 tx=0x7fa90c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ff9f4c2468202d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:17.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.001+0000 7fa911600640 1 -- 172.21.15.92:0/2559992885 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039070 con 0x7fa9140c60d0 2024-03-19T03:03:17.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.001+0000 7fa911600640 1 -- 172.21.15.92:0/2559992885 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c02cbf0 con 0x7fa9140c60d0 2024-03-19T03:03:17.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.001+0000 7fa911600640 1 -- 172.21.15.92:0/2559992885 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c034440 con 0x7fa9140c60d0 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 -- 172.21.15.92:0/2559992885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9140c60d0 msgr2=0x7fa9140c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 --2- 172.21.15.92:0/2559992885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9140c60d0 0x7fa9140c6c90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa914075340 tx=0x7fa90c02be40 comp rx=0 tx=0).stop 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 -- 172.21.15.92:0/2559992885 shutdown_connections 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 --2- 172.21.15.92:0/2559992885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9140c60d0 0x7fa9140c6c90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 --2- 172.21.15.92:0/2559992885 >> v2:172.21.15.92:3300/0 conn(0x7fa914072380 0x7fa914071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 -- 172.21.15.92:0/2559992885 >> 172.21.15.92:0/2559992885 conn(0x7fa9140c8d90 msgr2=0x7fa9140c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 -- 172.21.15.92:0/2559992885 shutdown_connections 2024-03-19T03:03:17.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.002+0000 7fa919600640 1 -- 172.21.15.92:0/2559992885 wait complete. 2024-03-19T03:03:17.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.005+0000 7fa919600640 1 Processor -- start 2024-03-19T03:03:17.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.005+0000 7fa919600640 1 -- start start 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9140c60d0 0x7fa914148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa914154dc0 0x7fa914149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa9140c7350 con 0x7fa914072380 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa9140c71d0 con 0x7fa914154dc0 2024-03-19T03:03:17.008 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa919600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa9140c74d0 con 0x7fa9140c60d0 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa912a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa912a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49206/0 (socket says 172.21.15.92:49206) 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa912a00640 1 -- 172.21.15.92:0/2477735948 learned_addr learned my addr 172.21.15.92:0/2477735948 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.006+0000 7fa913400640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa914154dc0 0x7fa914149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 -- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9140c60d0 msgr2=0x7fa914148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9140c60d0 0x7fa914148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 -- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa914154dc0 msgr2=0x7fa914149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa914154dc0 0x7fa914149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 -- 172.21.15.92:0/2477735948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91415bea0 con 0x7fa914072380 2024-03-19T03:03:17.009 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa912a00640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c430 tx=0x7fa90400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:17.010 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9040074e0 con 0x7fa914072380 2024-03-19T03:03:17.010 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9040047e0 con 0x7fa914072380 2024-03-19T03:03:17.010 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.007+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91415c180 con 0x7fa914072380 2024-03-19T03:03:17.010 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.008+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904004bd0 con 0x7fa914072380 2024-03-19T03:03:17.010 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.008+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91415c740 con 0x7fa914072380 2024-03-19T03:03:17.011 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.009+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9140c57f0 con 0x7fa914072380 2024-03-19T03:03:17.011 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.010+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa904011070 con 0x7fa914072380 2024-03-19T03:03:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.011+0000 7fa90aa00640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 0x7fa8f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.011+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7fa90408a860 con 0x7fa914072380 2024-03-19T03:03:17.013 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.011+0000 7fa912000640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 0x7fa8f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:17.013 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.012+0000 7fa912000640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 0x7fa8f006e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006a90 tx=0x7fa8fc006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:17.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.016+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa9140c57f0 con 0x7fa914072380 2024-03-19T03:03:17.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.247+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa914067e50 con 0x7fa914072380 2024-03-19T03:03:17.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.248+0000 7fa90aa00640 1 -- 172.21.15.92:0/2477735948 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7fa904059140 con 0x7fa914072380 2024-03-19T03:03:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:17.250 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:17.250 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 msgr2=0x7fa8f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 0x7fa8f006e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006a90 tx=0x7fa8fc006a20 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 msgr2=0x7fa91415e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c430 tx=0x7fa90400c8f0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 shutdown_connections 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa8f006bbd0 0x7fa8f006e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa914154dc0 0x7fa914149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9140c60d0 0x7fa914148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 --2- 172.21.15.92:0/2477735948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa914072380 0x7fa91415e270 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 >> 172.21.15.92:0/2477735948 conn(0x7fa9140c8d90 msgr2=0x7fa914072a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 shutdown_connections 2024-03-19T03:03:17.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:17.251+0000 7fa919600640 1 -- 172.21.15.92:0/2477735948 wait complete. 2024-03-19T03:03:17.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:19.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:19.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.493+0000 7fedb8c00640 1 Processor -- start 2024-03-19T03:03:19.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.493+0000 7fedb8c00640 1 -- start start 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb8c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 0x7fedb4071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb8c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb415b740 con 0x7fedb4072380 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb8c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb415ba20 con 0x7fedb4154dc0 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb8c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fedb40c8400 con 0x7fedb40c60d0 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb1e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fedb4154dc0 0x7fedb4071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:42116/0 (socket says 172.21.15.92:42116) 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb1400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fedb4072380 0x7fedb40c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48012/0 (socket says 172.21.15.92:48012) 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb1e00640 1 -- 172.21.15.92:0/888683009 learned_addr learned my addr 172.21.15.92:0/888683009 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:19.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.495+0000 7fedb2800640 1 --2- 172.21.15.92:0/888683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 0x7fedb4071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:19.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.496+0000 7fedb0a00640 1 -- 172.21.15.92:0/888683009 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538799627 0 0) 0x7fedb415ba20 con 0x7fedb4154dc0 2024-03-19T03:03:19.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.496+0000 7fedb0a00640 1 -- 172.21.15.92:0/888683009 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed98003520 con 0x7fedb4154dc0 2024-03-19T03:03:19.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.496+0000 7fedb2800640 1 -- 172.21.15.92:0/888683009 >> v1:172.21.15.159:6789/0 conn(0x7fedb4154dc0 legacy=0x7fedb4071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.496+0000 7fedb2800640 1 -- 172.21.15.92:0/888683009 >> v1:172.21.15.92:6789/0 conn(0x7fedb4072380 legacy=0x7fedb40c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.496+0000 7fedb2800640 1 -- 172.21.15.92:0/888683009 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb40c86e0 con 0x7fedb40c60d0 2024-03-19T03:03:19.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.497+0000 7fedb2800640 1 --2- 172.21.15.92:0/888683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 0x7fedb4071190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fedac009dc0 tx=0x7fedac02be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71e5d0690615b66d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:19.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.497+0000 7fedb0a00640 1 -- 172.21.15.92:0/888683009 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac039070 con 0x7fedb40c60d0 2024-03-19T03:03:19.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.497+0000 7fedb0a00640 1 -- 172.21.15.92:0/888683009 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedac02cc30 con 0x7fedb40c60d0 2024-03-19T03:03:19.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.497+0000 7fedb0a00640 1 -- 172.21.15.92:0/888683009 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac034440 con 0x7fedb40c60d0 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 -- 172.21.15.92:0/888683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 msgr2=0x7fedb4071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 --2- 172.21.15.92:0/888683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 0x7fedb4071190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fedac009dc0 tx=0x7fedac02be80 comp rx=0 tx=0).stop 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 -- 172.21.15.92:0/888683009 shutdown_connections 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 --2- 172.21.15.92:0/888683009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb40c60d0 0x7fedb4071190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 -- 172.21.15.92:0/888683009 >> 172.21.15.92:0/888683009 conn(0x7fedb40c8d90 msgr2=0x7fedb40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 -- 172.21.15.92:0/888683009 shutdown_connections 2024-03-19T03:03:19.499 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.498+0000 7fedb8c00640 1 -- 172.21.15.92:0/888683009 wait complete. 2024-03-19T03:03:19.500 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.500+0000 7fedb8c00640 1 Processor -- start 2024-03-19T03:03:19.500 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.500+0000 7fedb8c00640 1 -- start start 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb4072380 0x7fedb41f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedb4154dc0 0x7fedb41e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fedb40c7520 con 0x7fedb40c60d0 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fedb40c73a0 con 0x7fedb4154dc0 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb8c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fedb40c76a0 con 0x7fedb4072380 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49212/0 (socket says 172.21.15.92:49212) 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb1400640 1 -- 172.21.15.92:0/3540245677 learned_addr learned my addr 172.21.15.92:0/3540245677 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb2800640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedb4154dc0 0x7fedb41e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.501+0000 7fedb1e00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb4072380 0x7fedb41f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 -- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb4072380 msgr2=0x7fedb41f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.503 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb4072380 0x7fedb41f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 -- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedb4154dc0 msgr2=0x7fedb41e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedb4154dc0 0x7fedb41e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 -- 172.21.15.92:0/3540245677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb41e5020 con 0x7fedb40c60d0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb1400640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feda400c430 tx=0x7feda400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda40074e0 con 0x7fedb40c60d0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feda4007680 con 0x7fedb40c60d0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4004a00 con 0x7fedb40c60d0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb41e5300 con 0x7fedb40c60d0 2024-03-19T03:03:19.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.502+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41e58c0 con 0x7fedb40c60d0 2024-03-19T03:03:19.505 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.503+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb40684c0 con 0x7fedb40c60d0 2024-03-19T03:03:19.505 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.504+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feda401a020 con 0x7fedb40c60d0 2024-03-19T03:03:19.506 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.505+0000 7fedaa000640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 0x7fed9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:19.506 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.505+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7feda408a2d0 con 0x7fedb40c60d0 2024-03-19T03:03:19.506 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.506+0000 7fedb1e00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 0x7fed9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:19.506 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.506+0000 7fedb1e00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 0x7fed9006e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0011e0 tx=0x7fed9c007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:19.510 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.510+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feda4058570 con 0x7fedb40c60d0 2024-03-19T03:03:19.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.795+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedb4073510 con 0x7fedb40c60d0 2024-03-19T03:03:19.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.796+0000 7fedaa000640 1 -- 172.21.15.92:0/3540245677 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7feda405c610 con 0x7fedb40c60d0 2024-03-19T03:03:19.797 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:19.797 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:19.797 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:19.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 msgr2=0x7fed9006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 0x7fed9006e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0011e0 tx=0x7fed9c007d30 comp rx=0 tx=0).stop 2024-03-19T03:03:19.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 msgr2=0x7fedb40ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:19.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feda400c430 tx=0x7feda400c8f0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.800 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 shutdown_connections 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fed9006bc00 0x7fed9006e0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedb4154dc0 0x7fedb41e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb40c60d0 0x7fedb40ca3c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 --2- 172.21.15.92:0/3540245677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedb4072380 0x7fedb41f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 >> 172.21.15.92:0/3540245677 conn(0x7fedb40c8d90 msgr2=0x7fedb406e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 shutdown_connections 2024-03-19T03:03:19.801 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:19.799+0000 7fedb8c00640 1 -- 172.21.15.92:0/3540245677 wait complete. 2024-03-19T03:03:19.817 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:21.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:22.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.036+0000 7f6439800640 1 Processor -- start 2024-03-19T03:03:22.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.036+0000 7f6439800640 1 -- start start 2024-03-19T03:03:22.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.037+0000 7f6439800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64340c6470 0x7f64340c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.037+0000 7f6439800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f64341593d0 0x7f64340c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.037+0000 7f6439800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64340c9a80 con 0x7f6434152bf0 2024-03-19T03:03:22.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.037+0000 7f6439800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f64340c6c90 con 0x7f64341593d0 2024-03-19T03:03:22.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.037+0000 7f6439800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6434159910 con 0x7f64340c6470 2024-03-19T03:03:22.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6432a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6434152bf0 0x7f643406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48028/0 (socket says 172.21.15.92:48028) 2024-03-19T03:03:22.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6432a00640 1 -- 172.21.15.92:0/125640663 learned_addr learned my addr 172.21.15.92:0/125640663 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:22.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6433400640 1 --2- 172.21.15.92:0/125640663 >> v2:172.21.15.92:3300/0 conn(0x7f64341593d0 0x7f64340c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6432000640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64340c6470 0x7f64340c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6431600640 1 -- 172.21.15.92:0/125640663 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402996772 0 0) 0x7f64340c9a80 con 0x7f6434152bf0 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6431600640 1 -- 172.21.15.92:0/125640663 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6418003520 con 0x7f6434152bf0 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6433400640 1 -- 172.21.15.92:0/125640663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64340c6470 msgr2=0x7f64340c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6433400640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64340c6470 0x7f64340c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6433400640 1 -- 172.21.15.92:0/125640663 >> v1:172.21.15.92:6789/0 conn(0x7f6434152bf0 legacy=0x7f643406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.038+0000 7f6433400640 1 -- 172.21.15.92:0/125640663 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64340c8650 con 0x7f64341593d0 2024-03-19T03:03:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.039+0000 7f6433400640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64341593d0 0x7f64340c44c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f642c0097a0 tx=0x7f642c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a733b27d494a47d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:22.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6431600640 1 -- 172.21.15.92:0/125640663 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c0315a0 con 0x7f64341593d0 2024-03-19T03:03:22.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6431600640 1 -- 172.21.15.92:0/125640663 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642c037070 con 0x7f64341593d0 2024-03-19T03:03:22.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6431600640 1 -- 172.21.15.92:0/125640663 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c031960 con 0x7f64341593d0 2024-03-19T03:03:22.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 -- 172.21.15.92:0/125640663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64341593d0 msgr2=0x7f64340c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64341593d0 0x7f64340c44c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f642c0097a0 tx=0x7f642c02b840 comp rx=0 tx=0).stop 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 -- 172.21.15.92:0/125640663 shutdown_connections 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64341593d0 0x7f64340c44c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 --2- 172.21.15.92:0/125640663 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64340c6470 0x7f64340c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 -- 172.21.15.92:0/125640663 >> 172.21.15.92:0/125640663 conn(0x7f64340c8d90 msgr2=0x7f64340c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 -- 172.21.15.92:0/125640663 shutdown_connections 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.040+0000 7f6439800640 1 -- 172.21.15.92:0/125640663 wait complete. 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.042+0000 7f6439800640 1 Processor -- start 2024-03-19T03:03:22.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.042+0000 7f6439800640 1 -- start start 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 0x7f64341e4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 0x7f6434145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64341593d0 0x7f64341464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f643406ea60 con 0x7f6434152bf0 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f643406e4a0 con 0x7f64340c6470 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6439800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f643406e6f0 con 0x7f64341593d0 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6432000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 0x7f6434145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6432a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 0x7f64341e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6432000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 0x7f6434145f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49228/0 (socket says 172.21.15.92:49228) 2024-03-19T03:03:22.045 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.043+0000 7f6432000640 1 -- 172.21.15.92:0/2390458466 learned_addr learned my addr 172.21.15.92:0/2390458466 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:22.045 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6433400640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64341593d0 0x7f64341464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 -- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64341593d0 msgr2=0x7f64341464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64341593d0 0x7f64341464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 -- 172.21.15.92:0/2390458466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 msgr2=0x7f6434145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 0x7f6434145f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 -- 172.21.15.92:0/2390458466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434156940 con 0x7f64340c6470 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.044+0000 7f6432a00640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 0x7f64341e4280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f642400ee30 tx=0x7f6424010420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.045+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424010bf0 con 0x7f64340c6470 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.045+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6424010d90 con 0x7f64340c6470 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.045+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642400c670 con 0x7f64340c6470 2024-03-19T03:03:22.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.045+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64340ca400 con 0x7f64340c6470 2024-03-19T03:03:22.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.045+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64340ca9c0 con 0x7f64340c6470 2024-03-19T03:03:22.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.046+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64340684c0 con 0x7f64340c6470 2024-03-19T03:03:22.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.047+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f642400c810 con 0x7f64340c6470 2024-03-19T03:03:22.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.048+0000 7f642aa00640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 0x7f641406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:22.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.048+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f642408eff0 con 0x7f64340c6470 2024-03-19T03:03:22.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.049+0000 7f6432000640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 0x7f641406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:22.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.049+0000 7f6432000640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 0x7f641406e0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f643415cf30 tx=0x7f6420009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:22.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.053+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f642405d210 con 0x7f64340c6470 2024-03-19T03:03:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.293+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64340c3b50 con 0x7f64340c6470 2024-03-19T03:03:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.294+0000 7f642aa00640 1 -- 172.21.15.92:0/2390458466 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6839 (secure 0 0 0) 0x7f64240612b0 con 0x7f64340c6470 2024-03-19T03:03:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:22.295 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:01:17.972117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"f","rank":0,"incarnation":21,"state":"up:active","state_seq":23,"addr":"172.21.15.159:6837/2066707194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":2066707194},{"type":"v1","addr":"172.21.15.159:6837","nonce":2066707194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5041,"qdb_cluster":[5041]},"id":1}]} 2024-03-19T03:03:22.296 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 25 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 msgr2=0x7f641406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 0x7f641406e0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f643415cf30 tx=0x7f6420009190 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 msgr2=0x7f64341e4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 0x7f64341e4280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f642400ee30 tx=0x7f6424010420 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 shutdown_connections 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f641406bc00 0x7f641406e0c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64341593d0 0x7f64341464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6434152bf0 0x7f6434145f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 --2- 172.21.15.92:0/2390458466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64340c6470 0x7f64341e4280 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.297+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 >> 172.21.15.92:0/2390458466 conn(0x7f64340c8d90 msgr2=0x7f6434147eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.298+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 shutdown_connections 2024-03-19T03:03:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:22.298+0000 7f6439800640 1 -- 172.21.15.92:0/2390458466 wait complete. 2024-03-19T03:03:22.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-19T03:03:24.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:24.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.537+0000 7f99d0a00640 1 Processor -- start 2024-03-19T03:03:24.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.537+0000 7f99d0a00640 1 -- start start 2024-03-19T03:03:24.539 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.538+0000 7f99d0a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f99cc0c5b70 0x7f99cc15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.539 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.538+0000 7f99d0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.539 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99d0a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc0c6d00 con 0x7f99cc06d160 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99d0a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f99cc06df20 con 0x7f99cc0c5b70 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99d0a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f99cc06d610 con 0x7f99cc0c5f50 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:58812/0 (socket says 172.21.15.92:58812) 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 -- 172.21.15.92:0/2589324269 learned_addr learned my addr 172.21.15.92:0/2589324269 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99cb200640 1 --1- 172.21.15.92:0/2589324269 >> v1:172.21.15.159:6789/0 conn(0x7f99cc06d160 0x7f99cc152a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:42128/0 (socket says 172.21.15.92:42128) 2024-03-19T03:03:24.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99ca800640 1 --2- 172.21.15.92:0/2589324269 >> v2:172.21.15.159:3300/0 conn(0x7f99cc0c5b70 0x7f99cc15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 -- 172.21.15.92:0/2589324269 >> v2:172.21.15.159:3300/0 conn(0x7f99cc0c5b70 msgr2=0x7f99cc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:24.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 --2- 172.21.15.92:0/2589324269 >> v2:172.21.15.159:3300/0 conn(0x7f99cc0c5b70 0x7f99cc15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 -- 172.21.15.92:0/2589324269 >> v1:172.21.15.159:6789/0 conn(0x7f99cc06d160 legacy=0x7f99cc152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.539+0000 7f99c3e00640 1 -- 172.21.15.92:0/2589324269 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc0c6fa0 con 0x7f99cc0c5f50 2024-03-19T03:03:24.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.540+0000 7f99c3e00640 1 --2- 172.21.15.92:0/2589324269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009870 tx=0x7f99b402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53b7d8ffc7311679 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.541+0000 7f99c9e00640 1 -- 172.21.15.92:0/2589324269 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b402f830 con 0x7f99cc0c5f50 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.541+0000 7f99c9e00640 1 -- 172.21.15.92:0/2589324269 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b402f9d0 con 0x7f99cc0c5f50 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.541+0000 7f99c9e00640 1 -- 172.21.15.92:0/2589324269 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b402fda0 con 0x7f99cc0c5f50 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.541+0000 7f99d0a00640 1 -- 172.21.15.92:0/2589324269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 msgr2=0x7f99cc152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.541+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2589324269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009870 tx=0x7f99b402ba10 comp rx=0 tx=0).stop 2024-03-19T03:03:24.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 -- 172.21.15.92:0/2589324269 shutdown_connections 2024-03-19T03:03:24.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2589324269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc0c5f50 0x7f99cc152520 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2589324269 >> v2:172.21.15.159:3300/0 conn(0x7f99cc0c5b70 0x7f99cc15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 -- 172.21.15.92:0/2589324269 >> 172.21.15.92:0/2589324269 conn(0x7f99cc0cc1f0 msgr2=0x7f99cc0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:24.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 -- 172.21.15.92:0/2589324269 shutdown_connections 2024-03-19T03:03:24.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.542+0000 7f99d0a00640 1 -- 172.21.15.92:0/2589324269 wait complete. 2024-03-19T03:03:24.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.544+0000 7f99d0a00640 1 Processor -- start 2024-03-19T03:03:24.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.544+0000 7f99d0a00640 1 -- start start 2024-03-19T03:03:24.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc06d160 0x7f99cc14abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 0x7f99cc14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc0729b0 con 0x7f99cc0c5f50 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc06dbb0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99d0a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f99cc06e3b0 con 0x7f99cc06d160 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99cb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99cb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49236/0 (socket says 172.21.15.92:49236) 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99c3e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 0x7f99cc14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99cb200640 1 -- 172.21.15.92:0/2075949641 learned_addr learned my addr 172.21.15.92:0/2075949641 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.545+0000 7f99ca800640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc06d160 0x7f99cc14abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 -- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc06d160 msgr2=0x7f99cc14abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc06d160 0x7f99cc14abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 -- 172.21.15.92:0/2075949641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 msgr2=0x7f99cc1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 -- 172.21.15.92:0/2075949641 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc1eed30 con 0x7f99cc0c5b70 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99cb200640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c3e00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 0x7f99cc14a430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f99cc06e0a0 tx=0x7f99b4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4047070 con 0x7f99cc0c5b70 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b4037be0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.546+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4042480 con 0x7f99cc0c5b70 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.547+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc1eefd0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.547+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc148be0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.548+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99cc04f580 con 0x7f99cc0c5b70 2024-03-19T03:03:24.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.549+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f99b402d3c0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.550+0000 7f99c2a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 0x7f99a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.550+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6112+0+0 (secure 0 0 0) 0x7f99b40accb0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.550+0000 7f99ca800640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 0x7f99a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.551+0000 7f99ca800640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 0x7f99a806e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009b40 tx=0x7f99bc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:24.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.555+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f99b4033030 con 0x7f99cc0c5b70 2024-03-19T03:03:24.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.878+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99cc072120 con 0x7f99cc0c5b70 2024-03-19T03:03:24.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.878+0000 7f99c2a00640 1 -- 172.21.15.92:0/2075949641 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6003 (secure 0 0 0) 0x7f99b407afd0 con 0x7f99cc0c5b70 2024-03-19T03:03:24.879 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:24.879 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:24.698565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:24.880 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 26 2024-03-19T03:03:24.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 msgr2=0x7f99a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 0x7f99a806e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009b40 tx=0x7f99bc009190 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 msgr2=0x7f99cc14a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 0x7f99cc14a430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f99cc06e0a0 tx=0x7f99b4004730 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 shutdown_connections 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f99a806bbd0 0x7f99a806e090 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009b40 tx=0x7f99bc009190 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99cc0c5f50 0x7f99cc1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99cc0c5b70 0x7f99cc14a430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 --2- 172.21.15.92:0/2075949641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f99cc06d160 0x7f99cc14abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:24.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.882+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 >> 172.21.15.92:0/2075949641 conn(0x7f99cc0cc1f0 msgr2=0x7f99cc0c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:24.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.883+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 shutdown_connections 2024-03-19T03:03:24.884 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:24.883+0000 7f99d0a00640 1 -- 172.21.15.92:0/2075949641 wait complete. 2024-03-19T03:03:24.903 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-19T03:03:24.904 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-19T03:03:24.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:25.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.080+0000 7f5fd4c00640 1 Processor -- start 2024-03-19T03:03:25.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.080+0000 7f5fd4c00640 1 -- start start 2024-03-19T03:03:25.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fd4c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5fd00c60d0 0x7f5fd0071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:25.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fd4c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fd015b740 con 0x7f5fd0154dc0 2024-03-19T03:03:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fd4c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fd015ba20 con 0x7f5fd0072380 2024-03-19T03:03:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fd4c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd00c8400 con 0x7f5fd00c60d0 2024-03-19T03:03:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fcde00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5fd0154dc0 0x7f5fd0071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:48032/0 (socket says 172.21.15.92:48032) 2024-03-19T03:03:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fcde00640 1 -- 172.21.15.92:0/1377573683 learned_addr learned my addr 172.21.15.92:0/1377573683 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:25.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fce800640 1 --2- 172.21.15.92:0/1377573683 >> v2:172.21.15.159:3300/0 conn(0x7f5fd00c60d0 0x7f5fd0071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.082+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616933784 0 0) 0x7f5fd015b740 con 0x7f5fd0154dc0 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fb0003520 con 0x7f5fd0154dc0 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035421027 0 0) 0x7f5fd015ba20 con 0x7f5fd0072380 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd015b740 con 0x7f5fd0072380 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fce800640 1 -- 172.21.15.92:0/1377573683 >> v1:172.21.15.159:6789/0 conn(0x7f5fd0072380 legacy=0x7f5fd00c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fce800640 1 -- 172.21.15.92:0/1377573683 >> v1:172.21.15.92:6789/0 conn(0x7f5fd0154dc0 legacy=0x7f5fd0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.083+0000 7f5fce800640 1 -- 172.21.15.92:0/1377573683 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd00c86e0 con 0x7f5fd00c60d0 2024-03-19T03:03:25.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fce800640 1 --2- 172.21.15.92:0/1377573683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd00c60d0 0x7f5fd0071190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009d90 tx=0x7f5fc802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dfacd61e199daea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:25.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8039070 con 0x7f5fd00c60d0 2024-03-19T03:03:25.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc802cc60 con 0x7f5fd00c60d0 2024-03-19T03:03:25.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fcd400640 1 -- 172.21.15.92:0/1377573683 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8034660 con 0x7f5fd00c60d0 2024-03-19T03:03:25.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fd4c00640 1 -- 172.21.15.92:0/1377573683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd00c60d0 msgr2=0x7f5fd0071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.084+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/1377573683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd00c60d0 0x7f5fd0071190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009d90 tx=0x7f5fc802beb0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.085+0000 7f5fd4c00640 1 -- 172.21.15.92:0/1377573683 shutdown_connections 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.085+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/1377573683 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd00c60d0 0x7f5fd0071190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.085+0000 7f5fd4c00640 1 -- 172.21.15.92:0/1377573683 >> 172.21.15.92:0/1377573683 conn(0x7f5fd00c8d90 msgr2=0x7f5fd00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.085+0000 7f5fd4c00640 1 -- 172.21.15.92:0/1377573683 shutdown_connections 2024-03-19T03:03:25.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.085+0000 7f5fd4c00640 1 -- 172.21.15.92:0/1377573683 wait complete. 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.086+0000 7f5fd4c00640 1 Processor -- start 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.086+0000 7f5fd4c00640 1 -- start start 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd0072380 0x7f5fd015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd00c60d0 0x7f5fd0149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd00c7520 con 0x7f5fd0154dc0 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd00c73a0 con 0x7f5fd0072380 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.087+0000 7f5fd4c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd00c76a0 con 0x7f5fd00c60d0 2024-03-19T03:03:25.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:49244/0 (socket says 172.21.15.92:49244) 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 -- 172.21.15.92:0/2123191510 learned_addr learned my addr 172.21.15.92:0/2123191510 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fcde00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd0072380 0x7f5fd015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fc7e00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd00c60d0 0x7f5fd0149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 -- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd00c60d0 msgr2=0x7f5fd0149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd00c60d0 0x7f5fd0149030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 -- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd0072380 msgr2=0x7f5fd015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd0072380 0x7f5fd015e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 -- 172.21.15.92:0/2123191510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd0157cf0 con 0x7f5fd0154dc0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.088+0000 7f5fce800640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009d90 tx=0x7f5fc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.089+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8039050 con 0x7f5fd0154dc0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.089+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc8045070 con 0x7f5fd0154dc0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.089+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8009450 con 0x7f5fd0154dc0 2024-03-19T03:03:25.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.089+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd0153840 con 0x7f5fd0154dc0 2024-03-19T03:03:25.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.089+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd015eca0 con 0x7f5fd0154dc0 2024-03-19T03:03:25.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.090+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd0158780 con 0x7f5fd0154dc0 2024-03-19T03:03:25.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.091+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5fc803e850 con 0x7f5fd0154dc0 2024-03-19T03:03:25.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.092+0000 7f5fc6000640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 0x7f5fac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:25.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.092+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f5fc80ad980 con 0x7f5fd0154dc0 2024-03-19T03:03:25.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.092+0000 7f5fcde00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 0x7f5fac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:25.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.093+0000 7f5fcde00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 0x7f5fac06e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000f60 tx=0x7f5fc0007d70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:25.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.096+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5fd0158780 con 0x7f5fd0154dc0 2024-03-19T03:03:25.323 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.322+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fd00684c0 con 0x7f5fd0154dc0 2024-03-19T03:03:25.324 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.324+0000 7f5fc6000640 1 -- 172.21.15.92:0/2123191510 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6003 (secure 0 0 0) 0x7f5fc807bcc0 con 0x7f5fd0154dc0 2024-03-19T03:03:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:25.325 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:24.698565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:25.325 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 26 2024-03-19T03:03:25.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.326+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 msgr2=0x7f5fac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.326+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 0x7f5fac06e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000f60 tx=0x7f5fc0007d70 comp rx=0 tx=0).stop 2024-03-19T03:03:25.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.326+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 msgr2=0x7f5fd015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:25.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.326+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009d90 tx=0x7f5fc8004a40 comp rx=0 tx=0).stop 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 shutdown_connections 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5fac06bbd0 0x7f5fac06e090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd0154dc0 0x7f5fd015cf90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd00c60d0 0x7f5fd0149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 --2- 172.21.15.92:0/2123191510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd0072380 0x7f5fd015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 >> 172.21.15.92:0/2123191510 conn(0x7f5fd00c8d90 msgr2=0x7f5fd00727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 shutdown_connections 2024-03-19T03:03:25.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:25.327+0000 7f5fd4c00640 1 -- 172.21.15.92:0/2123191510 wait complete. 2024-03-19T03:03:25.346 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5185": { "addr": "172.21.15.159:6839/3845908330", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6838", "nonce": 3845908330, "type": "v2" }, { "addr": "172.21.15.159:6839", "nonce": 3845908330, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5185, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:03:24.698565+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5185 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:03:27.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:27.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.563+0000 7fc6f1c00640 1 Processor -- start 2024-03-19T03:03:27.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.563+0000 7fc6f1c00640 1 -- start start 2024-03-19T03:03:27.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6f1c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec151900 0x7fc6ec158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6f1c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6ec151ce0 0x7fc6ec0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6f1c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ec0c3b80 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6f1c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ec158860 con 0x7fc6ec151ce0 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6f1c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec072e70 con 0x7fc6ec151900 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6eb400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc6ec0c7da0 0x7fc6ec15adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34654/0 (socket says 172.21.15.92:34654) 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6ea000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6ec151ce0 0x7fc6ec0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6eb400640 1 -- 172.21.15.92:0/1136820455 learned_addr learned my addr 172.21.15.92:0/1136820455 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:27.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.565+0000 7fc6eaa00640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec151900 0x7fc6ec158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.566+0000 7fc6ea000640 1 -- 172.21.15.92:0/1136820455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec151900 msgr2=0x7fc6ec158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.566+0000 7fc6ea000640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec151900 0x7fc6ec158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.566+0000 7fc6ea000640 1 -- 172.21.15.92:0/1136820455 >> v1:172.21.15.92:6789/0 conn(0x7fc6ec0c7da0 legacy=0x7fc6ec15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.566+0000 7fc6ea000640 1 -- 172.21.15.92:0/1136820455 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec073150 con 0x7fc6ec151ce0 2024-03-19T03:03:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.566+0000 7fc6e9600640 1 -- 172.21.15.92:0/1136820455 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952328352 0 0) 0x7fc6ec0c3b80 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.567+0000 7fc6ea000640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec0723d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80097a0 tx=0x7fc6d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf5e8bab52d566a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:27.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.567+0000 7fc6e9600640 1 -- 172.21.15.92:0/1136820455 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8034070 con 0x7fc6ec151ce0 2024-03-19T03:03:27.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.567+0000 7fc6e9600640 1 -- 172.21.15.92:0/1136820455 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d8030610 con 0x7fc6ec151ce0 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.567+0000 7fc6e9600640 1 -- 172.21.15.92:0/1136820455 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80309e0 con 0x7fc6ec151ce0 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1136820455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151ce0 msgr2=0x7fc6ec0723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec0723d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80097a0 tx=0x7fc6d802b960 comp rx=0 tx=0).stop 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1136820455 shutdown_connections 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec0723d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1136820455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec151900 0x7fc6ec158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1136820455 >> 172.21.15.92:0/1136820455 conn(0x7fc6ec0c8d90 msgr2=0x7fc6ec0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1136820455 shutdown_connections 2024-03-19T03:03:27.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.568+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1136820455 wait complete. 2024-03-19T03:03:27.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.570+0000 7fc6f1c00640 1 Processor -- start 2024-03-19T03:03:27.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.570+0000 7fc6f1c00640 1 -- start start 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 0x7fc6ec14da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec1f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec072910 con 0x7fc6ec151900 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec071eb0 con 0x7fc6ec151ce0 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6f1c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec0c9af0 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6ea000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6eaa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 0x7fc6ec14da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6ea000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:39006/0 (socket says 172.21.15.92:39006) 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6ea000640 1 -- 172.21.15.92:0/1287856175 learned_addr learned my addr 172.21.15.92:0/1287856175 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.571+0000 7fc6eb400640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec1f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 -- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6ec151ce0 msgr2=0x7fc6ec1f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec1f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 -- 172.21.15.92:0/1287856175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 msgr2=0x7fc6ec1f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 -- 172.21.15.92:0/1287856175 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec14e3d0 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6ea000640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:27.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6eaa00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 0x7fc6ec14da90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00e850 tx=0x7fc6dc00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc017070 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00c940 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00cd10 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec14e690 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.572+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1f3f90 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.573+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec0684c0 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.574+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc6dc01a070 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.575+0000 7fc6e2a00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 0x7fc6cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:27.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.575+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fc6dc08af90 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.576+0000 7fc6ea000640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 0x7fc6cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:27.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.576+0000 7fc6ea000640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 0x7fc6cc06e0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec1ee370 tx=0x7fc6d8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:27.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.580+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc6dc059110 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.817+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6ec0ca7b0 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.819 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.818+0000 7fc6e2a00640 1 -- 172.21.15.92:0/1287856175 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6003 (secure 0 0 0) 0x7fc6dc05d1b0 con 0x7fc6ec0c7da0 2024-03-19T03:03:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:27.820 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:24.698565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:27.820 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 26 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 msgr2=0x7fc6cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 0x7fc6cc06e0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec1ee370 tx=0x7fc6d8008030 comp rx=0 tx=0).stop 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 msgr2=0x7fc6ec14da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 0x7fc6ec14da90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00e850 tx=0x7fc6dc00ed10 comp rx=0 tx=0).stop 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 shutdown_connections 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc6cc06bc00 0x7fc6cc06e0c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6ec151ce0 0x7fc6ec1f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec151900 0x7fc6ec1f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 --2- 172.21.15.92:0/1287856175 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6ec0c7da0 0x7fc6ec14da90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:27.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.821+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 >> 172.21.15.92:0/1287856175 conn(0x7fc6ec0c8d90 msgr2=0x7fc6ec150130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:27.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.822+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 shutdown_connections 2024-03-19T03:03:27.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:27.822+0000 7fc6f1c00640 1 -- 172.21.15.92:0/1287856175 wait complete. 2024-03-19T03:03:27.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:29.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:30.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.052+0000 7f2f7fe00640 1 Processor -- start 2024-03-19T03:03:30.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.052+0000 7f2f7fe00640 1 -- start start 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7fe00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2f800c5480 0x7f2f80072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7fe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f80072140 con 0x7f2f8015a0c0 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7fe00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f800c4ef0 con 0x7f2f800c5860 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7fe00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f2f80159520 con 0x7f2f800c5480 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2f8015a0c0 0x7f2f80151610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34658/0 (socket says 172.21.15.92:34658) 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7f400640 1 -- 172.21.15.92:0/3660343755 learned_addr learned my addr 172.21.15.92:0/3660343755 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f7ea00640 1 --2- 172.21.15.92:0/3660343755 >> v2:172.21.15.159:3300/0 conn(0x7f2f800c5480 0x7f2f80072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.053+0000 7f2f77e00640 1 --1- 172.21.15.92:0/3660343755 >> v1:172.21.15.159:6789/0 conn(0x7f2f800c5860 0x7f2f801599b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:51212/0 (socket says 172.21.15.92:51212) 2024-03-19T03:03:30.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.054+0000 7f2f7ea00640 1 -- 172.21.15.92:0/3660343755 >> v1:172.21.15.159:6789/0 conn(0x7f2f800c5860 legacy=0x7f2f801599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.054+0000 7f2f7e000640 1 -- 172.21.15.92:0/3660343755 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636781941 0 0) 0x7f2f68001110 con 0x7f2f800c5860 2024-03-19T03:03:30.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.054+0000 7f2f7ea00640 1 -- 172.21.15.92:0/3660343755 >> v1:172.21.15.92:6789/0 conn(0x7f2f8015a0c0 legacy=0x7f2f80151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.054+0000 7f2f7ea00640 1 -- 172.21.15.92:0/3660343755 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f800728d0 con 0x7f2f800c5480 2024-03-19T03:03:30.057 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.055+0000 7f2f7ea00640 1 --2- 172.21.15.92:0/3660343755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80072390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2f700097a0 tx=0x7f2f7002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96fff4b8a0398d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:30.057 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.055+0000 7f2f7e000640 1 -- 172.21.15.92:0/3660343755 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70037070 con 0x7f2f800c5480 2024-03-19T03:03:30.057 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.055+0000 7f2f7e000640 1 -- 172.21.15.92:0/3660343755 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f70030710 con 0x7f2f800c5480 2024-03-19T03:03:30.057 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.055+0000 7f2f7e000640 1 -- 172.21.15.92:0/3660343755 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70030ae0 con 0x7f2f800c5480 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 -- 172.21.15.92:0/3660343755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 msgr2=0x7f2f80072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/3660343755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80072390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2f700097a0 tx=0x7f2f7002b710 comp rx=0 tx=0).stop 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 -- 172.21.15.92:0/3660343755 shutdown_connections 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/3660343755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80072390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 -- 172.21.15.92:0/3660343755 >> 172.21.15.92:0/3660343755 conn(0x7f2f800cc1f0 msgr2=0x7f2f800cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 -- 172.21.15.92:0/3660343755 shutdown_connections 2024-03-19T03:03:30.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.056+0000 7f2f7fe00640 1 -- 172.21.15.92:0/3660343755 wait complete. 2024-03-19T03:03:30.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.058+0000 7f2f7fe00640 1 Processor -- start 2024-03-19T03:03:30.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.058+0000 7f2f7fe00640 1 -- start start 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 0x7f2f801503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f8015a0c0 0x7f2f80150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80158ed0 con 0x7f2f800c5860 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80071e60 con 0x7f2f800c5480 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7fe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2f801591b0 con 0x7f2f8015a0c0 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7ea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:33238/0 (socket says 172.21.15.92:33238) 2024-03-19T03:03:30.062 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7ea00640 1 -- 172.21.15.92:0/2687935726 learned_addr learned my addr 172.21.15.92:0/2687935726 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f77e00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 0x7f2f801503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.059+0000 7f2f7f400640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f8015a0c0 0x7f2f80150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 -- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f8015a0c0 msgr2=0x7f2f80150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f8015a0c0 0x7f2f80150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 -- 172.21.15.92:0/2687935726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 msgr2=0x7f2f801503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 0x7f2f801503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 -- 172.21.15.92:0/2687935726 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f801f4430 con 0x7f2f800c5480 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f77e00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 0x7f2f801503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7ea00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f700097a0 tx=0x7f2f700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70042070 con 0x7f2f800c5480 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f70032a40 con 0x7f2f800c5480 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70037040 con 0x7f2f800c5480 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.060+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f801f4710 con 0x7f2f800c5480 2024-03-19T03:03:30.063 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.061+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f801f4cd0 con 0x7f2f800c5480 2024-03-19T03:03:30.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.062+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f8004f580 con 0x7f2f800c5480 2024-03-19T03:03:30.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.063+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2f70045070 con 0x7f2f800c5480 2024-03-19T03:03:30.065 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.063+0000 7f2f77400640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 0x7f2f5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.065 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.064+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f2f700aca40 con 0x7f2f800c5480 2024-03-19T03:03:30.065 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.064+0000 7f2f77e00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 0x7f2f5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.066 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.064+0000 7f2f77e00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 0x7f2f5c06e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f80071910 tx=0x7f2f68007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:30.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.068+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2f7007ae10 con 0x7f2f800c5480 2024-03-19T03:03:30.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.306+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f80159b40 con 0x7f2f800c5480 2024-03-19T03:03:30.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.307+0000 7f2f77400640 1 -- 172.21.15.92:0/2687935726 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6003 (secure 0 0 0) 0x7f2f7007eeb0 con 0x7f2f800c5480 2024-03-19T03:03:30.309 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:30.309 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:24.698565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:30.309 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 26 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 msgr2=0x7f2f5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 0x7f2f5c06e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f80071910 tx=0x7f2f68007b50 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 msgr2=0x7f2f80146f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f700097a0 tx=0x7f2f700046c0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 shutdown_connections 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f2f5c06bbd0 0x7f2f5c06e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f8015a0c0 0x7f2f80150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2f800c5860 0x7f2f801503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 --2- 172.21.15.92:0/2687935726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f800c5480 0x7f2f80146f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 >> 172.21.15.92:0/2687935726 conn(0x7f2f800cc1f0 msgr2=0x7f2f8014e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:30.312 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 shutdown_connections 2024-03-19T03:03:30.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:30.310+0000 7f2f7fe00640 1 -- 172.21.15.92:0/2687935726 wait complete. 2024-03-19T03:03:30.330 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:31.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:03:31.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:03:32.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:32.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.548+0000 7f9dabe00640 1 Processor -- start 2024-03-19T03:03:32.549 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.548+0000 7f9dabe00640 1 -- start start 2024-03-19T03:03:32.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.549+0000 7f9dabe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9dac15a0c0 0x7f9dac151610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.549+0000 7f9dabe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac072140 con 0x7f9dac0c5860 2024-03-19T03:03:32.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.549+0000 7f9dabe00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac0c4ef0 con 0x7f9dac0c5480 2024-03-19T03:03:32.551 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.549+0000 7f9dabe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9dac159520 con 0x7f9dac15a0c0 2024-03-19T03:03:32.552 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9daa000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9dac0c5860 0x7f9dac1599b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:34660/0 (socket says 172.21.15.92:34660) 2024-03-19T03:03:32.552 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9daa000640 1 -- 172.21.15.92:0/1512877136 learned_addr learned my addr 172.21.15.92:0/1512877136 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:32.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9daaa00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9dac0c5480 0x7f9dac072390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:51220/0 (socket says 172.21.15.92:51220) 2024-03-19T03:03:32.553 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9dab400640 1 --2- 172.21.15.92:0/1512877136 >> v2:172.21.15.92:3300/0 conn(0x7f9dac15a0c0 0x7f9dac151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2641556127 0 0) 0x7f9d9c001040 con 0x7f9dac0c5480 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d90003520 con 0x7f9dac0c5480 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1518925229 0 0) 0x7f9dac072140 con 0x7f9dac0c5860 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d9c001040 con 0x7f9dac0c5860 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9dab400640 1 -- 172.21.15.92:0/1512877136 >> v1:172.21.15.159:6789/0 conn(0x7f9dac0c5480 legacy=0x7f9dac072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9dab400640 1 -- 172.21.15.92:0/1512877136 >> v1:172.21.15.92:6789/0 conn(0x7f9dac0c5860 legacy=0x7f9dac1599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.550+0000 7f9dab400640 1 -- 172.21.15.92:0/1512877136 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac071b80 con 0x7f9dac15a0c0 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.551+0000 7f9dab400640 1 --2- 172.21.15.92:0/1512877136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac151610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009d90 tx=0x7f9da402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c15796c5b596663 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4039070 con 0x7f9dac15a0c0 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9da402cc60 con 0x7f9dac15a0c0 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9da9600640 1 -- 172.21.15.92:0/1512877136 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da4034660 con 0x7f9dac15a0c0 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 -- 172.21.15.92:0/1512877136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 msgr2=0x7f9dac151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 --2- 172.21.15.92:0/1512877136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac151610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009d90 tx=0x7f9da402beb0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 -- 172.21.15.92:0/1512877136 shutdown_connections 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 --2- 172.21.15.92:0/1512877136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac151610 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 -- 172.21.15.92:0/1512877136 >> 172.21.15.92:0/1512877136 conn(0x7f9dac0cc1f0 msgr2=0x7f9dac0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 -- 172.21.15.92:0/1512877136 shutdown_connections 2024-03-19T03:03:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.552+0000 7f9dabe00640 1 -- 172.21.15.92:0/1512877136 wait complete. 2024-03-19T03:03:32.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.554+0000 7f9dabe00640 1 Processor -- start 2024-03-19T03:03:32.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.554+0000 7f9dabe00640 1 -- start start 2024-03-19T03:03:32.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dac0c5860 0x7f9dac150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac1509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac1591b0 con 0x7f9dac15a0c0 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac158ed0 con 0x7f9dac0c5480 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.555+0000 7f9dabe00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9dac0c66a0 con 0x7f9dac0c5860 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9dab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac1509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:33246/0 (socket says 172.21.15.92:33246) 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 -- 172.21.15.92:0/3220241609 learned_addr learned my addr 172.21.15.92:0/3220241609 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daa000640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dac0c5860 0x7f9dac150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 -- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dac0c5860 msgr2=0x7f9dac150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dac0c5860 0x7f9dac150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 -- 172.21.15.92:0/3220241609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 msgr2=0x7f9dac1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac1509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 -- 172.21.15.92:0/3220241609 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac15aaa0 con 0x7f9dac0c5480 2024-03-19T03:03:32.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9dab400640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac1509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.556+0000 7f9daaa00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00c760 tx=0x7f9d9c00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.557+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c018070 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.557+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9c007480 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.557+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00ce10 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.557+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dac14ca30 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.557+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dac14c190 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.558+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dac04f580 con 0x7f9dac0c5480 2024-03-19T03:03:32.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.559+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d9c0047e0 con 0x7f9dac0c5480 2024-03-19T03:03:32.561 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.560+0000 7f9da2a00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 0x7f9d8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.561 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.560+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f9d9c08a140 con 0x7f9dac0c5480 2024-03-19T03:03:32.561 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.560+0000 7f9daa000640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 0x7f9d8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.561+0000 7f9daa000640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 0x7f9d8806e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0c44c0 tx=0x7f9d94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.564+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9dac0c4ef0 con 0x7f9dac0c5480 2024-03-19T03:03:32.804 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.802+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dac0c3bd0 con 0x7f9dac0c5480 2024-03-19T03:03:32.805 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.803+0000 7f9da2a00640 1 -- 172.21.15.92:0/3220241609 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6003 (secure 0 0 0) 0x7f9d9c0582c0 con 0x7f9dac0c5480 2024-03-19T03:03:32.805 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:32.805 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:24.698565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:32.805 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 26 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 msgr2=0x7f9d8806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 0x7f9d8806e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0c44c0 tx=0x7f9d94009040 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 msgr2=0x7f9dac13cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00c760 tx=0x7f9d9c00cc20 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 shutdown_connections 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f9d8806bc00 0x7f9d8806e0c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac15a0c0 0x7f9dac1509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9dac0c5860 0x7f9dac150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 --2- 172.21.15.92:0/3220241609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9dac0c5480 0x7f9dac13cd40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 >> 172.21.15.92:0/3220241609 conn(0x7f9dac0cc1f0 msgr2=0x7f9dac14df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 shutdown_connections 2024-03-19T03:03:32.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:32.807+0000 7f9dabe00640 1 -- 172.21.15.92:0/3220241609 wait complete. 2024-03-19T03:03:32.826 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:34.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:35.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.093+0000 7f598e200640 1 Processor -- start 2024-03-19T03:03:35.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.093+0000 7f598e200640 1 -- start start 2024-03-19T03:03:35.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.094+0000 7f598e200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5988151900 0x7f5988158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f598e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f598e200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880c3b80 con 0x7f59880c7da0 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f598e200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5988158860 con 0x7f5988151900 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f598e200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5988072e70 con 0x7f5988151ce0 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f5987400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5988151900 0x7f5988158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:47870/0 (socket says 172.21.15.92:47870) 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 -- 172.21.15.92:0/2327314810 learned_addr learned my addr 172.21.15.92:0/2327314810 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 -- 172.21.15.92:0/2327314810 >> v2:172.21.15.159:3300/0 conn(0x7f5988151900 msgr2=0x7f5988158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 --2- 172.21.15.92:0/2327314810 >> v2:172.21.15.159:3300/0 conn(0x7f5988151900 0x7f5988158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 -- 172.21.15.92:0/2327314810 >> v1:172.21.15.92:6789/0 conn(0x7f59880c7da0 legacy=0x7f598815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.095+0000 7f597ea00640 1 -- 172.21.15.92:0/2327314810 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5988073150 con 0x7f5988151ce0 2024-03-19T03:03:35.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.096+0000 7f597ea00640 1 --2- 172.21.15.92:0/2327314810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5970009870 tx=0x7f597002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e8a827c54fd845 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:35.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f5986a00640 1 -- 172.21.15.92:0/2327314810 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597002f830 con 0x7f5988151ce0 2024-03-19T03:03:35.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f5986a00640 1 -- 172.21.15.92:0/2327314810 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f597002f9d0 con 0x7f5988151ce0 2024-03-19T03:03:35.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f5986a00640 1 -- 172.21.15.92:0/2327314810 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597002fda0 con 0x7f5988151ce0 2024-03-19T03:03:35.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 -- 172.21.15.92:0/2327314810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 msgr2=0x7f59880723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 --2- 172.21.15.92:0/2327314810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5970009870 tx=0x7f597002ba10 comp rx=0 tx=0).stop 2024-03-19T03:03:35.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 -- 172.21.15.92:0/2327314810 shutdown_connections 2024-03-19T03:03:35.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 --2- 172.21.15.92:0/2327314810 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151ce0 0x7f59880723d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 --2- 172.21.15.92:0/2327314810 >> v2:172.21.15.159:3300/0 conn(0x7f5988151900 0x7f5988158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 -- 172.21.15.92:0/2327314810 >> 172.21.15.92:0/2327314810 conn(0x7f59880c8d90 msgr2=0x7f59880c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:35.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 -- 172.21.15.92:0/2327314810 shutdown_connections 2024-03-19T03:03:35.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.097+0000 7f598e200640 1 -- 172.21.15.92:0/2327314810 wait complete. 2024-03-19T03:03:35.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.099+0000 7f598e200640 1 Processor -- start 2024-03-19T03:03:35.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.099+0000 7f598e200640 1 -- start start 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 0x7f59881572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151900 0x7f5988154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 0x7f5988154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5988072910 con 0x7f5988151ce0 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5988071eb0 con 0x7f59880c7da0 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f598e200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59880c9af0 con 0x7f5988151900 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f5987400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 0x7f59881572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f5987e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 0x7f5988154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f5987400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 0x7f59881572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:33258/0 (socket says 172.21.15.92:33258) 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f5987400640 1 -- 172.21.15.92:0/1127950123 learned_addr learned my addr 172.21.15.92:0/1127950123 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:35.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.100+0000 7f597ea00640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151900 0x7f5988154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 -- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151900 msgr2=0x7f5988154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151900 0x7f5988154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 -- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 msgr2=0x7f59881572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 0x7f59881572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 -- 172.21.15.92:0/1127950123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880c3410 con 0x7f5988151ce0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f5987e00640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 0x7f5988154ad0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f598000c390 tx=0x7f598000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980017070 con 0x7f5988151ce0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59800052d0 con 0x7f5988151ce0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980003830 con 0x7f5988151ce0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.101+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59880c36b0 con 0x7f5988151ce0 2024-03-19T03:03:35.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.102+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5988152d10 con 0x7f5988151ce0 2024-03-19T03:03:35.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.103+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59881438a0 con 0x7f5988151ce0 2024-03-19T03:03:35.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.103+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f598001a070 con 0x7f5988151ce0 2024-03-19T03:03:35.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.104+0000 7f597fe00640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 0x7f596406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.104+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f5980089de0 con 0x7f5988151ce0 2024-03-19T03:03:35.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.105+0000 7f5987400640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 0x7f596406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.105+0000 7f5987400640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 0x7f596406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5978009a40 tx=0x7f597800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:35.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.109+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59881438a0 con 0x7f5988151ce0 2024-03-19T03:03:35.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.354+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59880684c0 con 0x7f5988151ce0 2024-03-19T03:03:35.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.355+0000 7f597fe00640 1 -- 172.21.15.92:0/1127950123 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6007 (secure 0 0 0) 0x7f5980058140 con 0x7f5988151ce0 2024-03-19T03:03:35.357 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:35.357 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:32.364482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:35.357 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 27 2024-03-19T03:03:35.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 msgr2=0x7f596406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 0x7f596406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5978009a40 tx=0x7f597800b040 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 msgr2=0x7f5988154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 0x7f5988154ad0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f598000c390 tx=0x7f598000c850 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 shutdown_connections 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f596406bbd0 0x7f596406e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5988151ce0 0x7f5988154ad0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5988151900 0x7f5988154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 --2- 172.21.15.92:0/1127950123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59880c7da0 0x7f59881572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 >> 172.21.15.92:0/1127950123 conn(0x7f59880c8d90 msgr2=0x7f5988157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 shutdown_connections 2024-03-19T03:03:35.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:35.358+0000 7f598e200640 1 -- 172.21.15.92:0/1127950123 wait complete. 2024-03-19T03:03:35.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:37.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:37.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.595+0000 7f9275a00640 1 Processor -- start 2024-03-19T03:03:37.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.596+0000 7f9275a00640 1 -- start start 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9270154de0 0x7f92700cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 0x7f927006bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f927006c520 0x7f92700cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f92700cccc0 con 0x7f927006c520 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f92700b3290 con 0x7f92700b34c0 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92700b2cf0 con 0x7f9270154de0 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f927006c520 0x7f92700cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f926f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 0x7f927006bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f927006c520 0x7f92700cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55054/0 (socket says 172.21.15.92:55054) 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f926fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9270154de0 0x7f92700cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.597+0000 7f9275000640 1 -- 172.21.15.92:0/1642207143 learned_addr learned my addr 172.21.15.92:0/1642207143 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f9275000640 1 -- 172.21.15.92:0/1642207143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9270154de0 msgr2=0x7f92700cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f9275000640 1 --2- 172.21.15.92:0/1642207143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9270154de0 0x7f92700cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f9275000640 1 -- 172.21.15.92:0/1642207143 >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 msgr2=0x7f927006bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f9275000640 1 --2- 172.21.15.92:0/1642207143 >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 0x7f927006bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f9275000640 1 -- 172.21.15.92:0/1642207143 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927015ef90 con 0x7f927006c520 2024-03-19T03:03:37.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.598+0000 7f926f400640 1 --2- 172.21.15.92:0/1642207143 >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 0x7f927006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:37.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.599+0000 7f9275000640 1 --2- 172.21.15.92:0/1642207143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cd1c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92700d1450 tx=0x7f926802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9447d9c3a13660a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:37.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f926ea00640 1 -- 172.21.15.92:0/1642207143 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926802ca70 con 0x7f927006c520 2024-03-19T03:03:37.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f926ea00640 1 -- 172.21.15.92:0/1642207143 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926802cc10 con 0x7f927006c520 2024-03-19T03:03:37.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f926ea00640 1 -- 172.21.15.92:0/1642207143 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680343f0 con 0x7f927006c520 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 -- 172.21.15.92:0/1642207143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 msgr2=0x7f92700cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 --2- 172.21.15.92:0/1642207143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cd1c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92700d1450 tx=0x7f926802bed0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 -- 172.21.15.92:0/1642207143 shutdown_connections 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 --2- 172.21.15.92:0/1642207143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cd1c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 --2- 172.21.15.92:0/1642207143 >> v2:172.21.15.159:3300/0 conn(0x7f92700b34c0 0x7f927006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 --2- 172.21.15.92:0/1642207143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9270154de0 0x7f92700cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 -- 172.21.15.92:0/1642207143 >> 172.21.15.92:0/1642207143 conn(0x7f9270067130 msgr2=0x7f9270067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:37.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.600+0000 7f9275a00640 1 -- 172.21.15.92:0/1642207143 shutdown_connections 2024-03-19T03:03:37.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.601+0000 7f9275a00640 1 -- 172.21.15.92:0/1642207143 wait complete. 2024-03-19T03:03:37.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.602+0000 7f9275a00640 1 Processor -- start 2024-03-19T03:03:37.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 -- start start 2024-03-19T03:03:37.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92700b34c0 0x7f92700c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9270154de0 0x7f92700c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f927015e4b0 con 0x7f927006c520 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f927015e1d0 con 0x7f9270154de0 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.603+0000 7f9275a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f927015e790 con 0x7f92700b34c0 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55070/0 (socket says 172.21.15.92:55070) 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 -- 172.21.15.92:0/1088707113 learned_addr learned my addr 172.21.15.92:0/1088707113 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926f400640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92700b34c0 0x7f92700c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f9275000640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9270154de0 0x7f92700c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 -- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92700b34c0 msgr2=0x7f92700c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92700b34c0 0x7f92700c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 -- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9270154de0 msgr2=0x7f92700c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9270154de0 0x7f92700c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.604+0000 7f926fe00640 1 -- 172.21.15.92:0/1088707113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92701f46d0 con 0x7f927006c520 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f926fe00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f926000e8a0 tx=0x7f926000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260017070 con 0x7f927006c520 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926000c940 con 0x7f927006c520 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926000cd30 con 0x7f927006c520 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92700cff50 con 0x7f927006c520 2024-03-19T03:03:37.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.605+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92701f49b0 con 0x7f927006c520 2024-03-19T03:03:37.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.606+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f927004f580 con 0x7f927006c520 2024-03-19T03:03:37.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.607+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f926001a070 con 0x7f927006c520 2024-03-19T03:03:37.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.608+0000 7f9267e00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 0x7f925006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:37.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.608+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f926008aea0 con 0x7f927006c520 2024-03-19T03:03:37.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.609+0000 7f926f400640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 0x7f925006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:37.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.609+0000 7f926f400640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 0x7f925006e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f925c009560 tx=0x7f925c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:37.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.613+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f92600590b0 con 0x7f927006c520 2024-03-19T03:03:37.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.848+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92700cbd20 con 0x7f927006c520 2024-03-19T03:03:37.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.849+0000 7f9267e00640 1 -- 172.21.15.92:0/1088707113 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6007 (secure 0 0 0) 0x7f926005d150 con 0x7f927006c520 2024-03-19T03:03:37.851 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:37.851 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:32.364482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:37.851 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 27 2024-03-19T03:03:37.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 msgr2=0x7f925006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 0x7f925006e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f925c009560 tx=0x7f925c009190 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 msgr2=0x7f92700cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f926000e8a0 tx=0x7f926000ed60 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 shutdown_connections 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f925006bc00 0x7f925006e0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9270154de0 0x7f92700c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92700b34c0 0x7f92700c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 --2- 172.21.15.92:0/1088707113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f927006c520 0x7f92700cf700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.851+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 >> 172.21.15.92:0/1088707113 conn(0x7f9270067130 msgr2=0x7f92700cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.852+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 shutdown_connections 2024-03-19T03:03:37.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:37.852+0000 7f9275a00640 1 -- 172.21.15.92:0/1088707113 wait complete. 2024-03-19T03:03:37.873 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5185": { "addr": "172.21.15.159:6839/3845908330", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6838", "nonce": 3845908330, "type": "v2" }, { "addr": "172.21.15.159:6839", "nonce": 3845908330, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5185, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:03:32.364482+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5185 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:03:39.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:40.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.089+0000 7f68aea00640 1 Processor -- start 2024-03-19T03:03:40.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.090+0000 7f68aea00640 1 -- start start 2024-03-19T03:03:40.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68aea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 0x7f68a8152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:40.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68aea00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a80c6d00 con 0x7f68a80c5b70 2024-03-19T03:03:40.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68aea00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a806d610 con 0x7f68a806d160 2024-03-19T03:03:40.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68aea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f68a815c580 con 0x7f68a80c5f50 2024-03-19T03:03:40.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68acc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 0x7f68a8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:40.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.092+0000 7f68ad600640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f68a80c5b70 0x7f68a815be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36706/0 (socket says 172.21.15.92:36706) 2024-03-19T03:03:40.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.093+0000 7f68ad600640 1 -- 172.21.15.92:0/2266918189 learned_addr learned my addr 172.21.15.92:0/2266918189 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:40.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.093+0000 7f68ae000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f68a806d160 0x7f68a8152a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52534/0 (socket says 172.21.15.92:52534) 2024-03-19T03:03:40.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.093+0000 7f68acc00640 1 -- 172.21.15.92:0/2266918189 >> v1:172.21.15.159:6789/0 conn(0x7f68a806d160 legacy=0x7f68a8152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.093+0000 7f68acc00640 1 -- 172.21.15.92:0/2266918189 >> v1:172.21.15.92:6789/0 conn(0x7f68a80c5b70 legacy=0x7f68a815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.094+0000 7f68acc00640 1 -- 172.21.15.92:0/2266918189 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a80c6fa0 con 0x7f68a80c5f50 2024-03-19T03:03:40.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.096+0000 7f68acc00640 1 --2- 172.21.15.92:0/2266918189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 0x7f68a8152520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097d0 tx=0x7f689c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a592a753ea7bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:40.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.097+0000 7f68a7e00640 1 -- 172.21.15.92:0/2266918189 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c034070 con 0x7f68a80c5f50 2024-03-19T03:03:40.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.098+0000 7f68a7e00640 1 -- 172.21.15.92:0/2266918189 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f689c030890 con 0x7f68a80c5f50 2024-03-19T03:03:40.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.098+0000 7f68a7e00640 1 -- 172.21.15.92:0/2266918189 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c030c60 con 0x7f68a80c5f50 2024-03-19T03:03:40.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.099+0000 7f68aea00640 1 -- 172.21.15.92:0/2266918189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 msgr2=0x7f68a8152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.099+0000 7f68aea00640 1 --2- 172.21.15.92:0/2266918189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 0x7f68a8152520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097d0 tx=0x7f689c02b9d0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.100+0000 7f68aea00640 1 -- 172.21.15.92:0/2266918189 shutdown_connections 2024-03-19T03:03:40.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.100+0000 7f68aea00640 1 --2- 172.21.15.92:0/2266918189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5f50 0x7f68a8152520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.100+0000 7f68aea00640 1 -- 172.21.15.92:0/2266918189 >> 172.21.15.92:0/2266918189 conn(0x7f68a80cc1f0 msgr2=0x7f68a80cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:40.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.100+0000 7f68aea00640 1 -- 172.21.15.92:0/2266918189 shutdown_connections 2024-03-19T03:03:40.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.100+0000 7f68aea00640 1 -- 172.21.15.92:0/2266918189 wait complete. 2024-03-19T03:03:40.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.106+0000 7f68aea00640 1 Processor -- start 2024-03-19T03:03:40.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.107+0000 7f68aea00640 1 -- start start 2024-03-19T03:03:40.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a806d160 0x7f68a81ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5b70 0x7f68a8147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f68a806dbb0 con 0x7f68a806d160 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f68a806df20 con 0x7f68a80c5f50 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68aea00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f68a80729b0 con 0x7f68a80c5b70 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68ae000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:40.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68ae000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:41838/0 (socket says 172.21.15.92:41838) 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68ae000640 1 -- 172.21.15.92:0/22874399 learned_addr learned my addr 172.21.15.92:0/22874399 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.110+0000 7f68ad600640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a806d160 0x7f68a81ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68acc00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5b70 0x7f68a8147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68ae000640 1 -- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5b70 msgr2=0x7f68a8147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68ae000640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5b70 0x7f68a8147d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68ae000640 1 -- 172.21.15.92:0/22874399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a806d160 msgr2=0x7f68a81ef790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68ae000640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a806d160 0x7f68a81ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.111+0000 7f68ae000640 1 -- 172.21.15.92:0/22874399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a81579c0 con 0x7f68a80c5f50 2024-03-19T03:03:40.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.112+0000 7f68ae000640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c710 tx=0x7f68a000cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:40.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.112+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0017070 con 0x7f68a80c5f50 2024-03-19T03:03:40.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.113+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a000ce20 con 0x7f68a80c5f50 2024-03-19T03:03:40.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.113+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0017070 con 0x7f68a80c5f50 2024-03-19T03:03:40.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.113+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a81566a0 con 0x7f68a80c5f50 2024-03-19T03:03:40.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.113+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a8156c40 con 0x7f68a80c5f50 2024-03-19T03:03:40.117 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.115+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a804f580 con 0x7f68a80c5f50 2024-03-19T03:03:40.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.118+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f68a001a070 con 0x7f68a80c5f50 2024-03-19T03:03:40.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.119+0000 7f68a5600640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 0x7f688806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:40.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.119+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f68a0089e60 con 0x7f68a80c5f50 2024-03-19T03:03:40.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.120+0000 7f68ad600640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 0x7f688806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:40.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.121+0000 7f68ad600640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 0x7f688806e2c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6894000f60 tx=0x7f6894009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:40.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.125+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f68a00587f0 con 0x7f68a80c5f50 2024-03-19T03:03:40.359 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.358+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68a814c5f0 con 0x7f68a80c5f50 2024-03-19T03:03:40.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.359+0000 7f68a5600640 1 -- 172.21.15.92:0/22874399 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6004 (secure 0 0 0) 0x7f68a00580f0 con 0x7f68a80c5f50 2024-03-19T03:03:40.361 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:40.361 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:38.079258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:40.361 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 28 2024-03-19T03:03:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.361+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 msgr2=0x7f688806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.361+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 0x7f688806e2c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6894000f60 tx=0x7f6894009040 comp rx=0 tx=0).stop 2024-03-19T03:03:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.361+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 msgr2=0x7f68a8145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.361+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c710 tx=0x7f68a000cbd0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 shutdown_connections 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f688806be00 0x7f688806e2c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68a80c5f50 0x7f68a8145ac0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68a80c5b70 0x7f68a8147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 --2- 172.21.15.92:0/22874399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a806d160 0x7f68a81ef790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 >> 172.21.15.92:0/22874399 conn(0x7f68a80cc1f0 msgr2=0x7f68a814a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 shutdown_connections 2024-03-19T03:03:40.364 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:40.362+0000 7f68aea00640 1 -- 172.21.15.92:0/22874399 wait complete. 2024-03-19T03:03:40.382 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:42.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:42.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.601+0000 7f6aee000640 1 Processor -- start 2024-03-19T03:03:42.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.601+0000 7f6aee000640 1 -- start start 2024-03-19T03:03:42.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6aee000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ae8072380 0x7f6ae8071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:42.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6aee000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae815b740 con 0x7f6ae80c60d0 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6aee000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae815ba20 con 0x7f6ae8154dc0 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6aee000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae80c8400 con 0x7f6ae8072380 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ae8072380 0x7f6ae8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ae8072380 0x7f6ae8071190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:55080/0 (socket says 172.21.15.92:55080) 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 -- 172.21.15.92:0/2239622864 learned_addr learned my addr 172.21.15.92:0/2239622864 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 -- 172.21.15.92:0/2239622864 >> v1:172.21.15.159:6789/0 conn(0x7f6ae8154dc0 legacy=0x7f6ae8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 -- 172.21.15.92:0/2239622864 >> v1:172.21.15.92:6789/0 conn(0x7f6ae80c60d0 legacy=0x7f6ae80c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.603+0000 7f6ae6a00640 1 -- 172.21.15.92:0/2239622864 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae80c86e0 con 0x7f6ae8072380 2024-03-19T03:03:42.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6ae6a00640 1 --2- 172.21.15.92:0/2239622864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8072380 0x7f6ae8071190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009870 tx=0x7f6ad002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb042151cc79dfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:42.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6ae6000640 1 -- 172.21.15.92:0/2239622864 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0035020 con 0x7f6ae8072380 2024-03-19T03:03:42.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6ae6000640 1 -- 172.21.15.92:0/2239622864 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad002d040 con 0x7f6ae8072380 2024-03-19T03:03:42.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6ae6000640 1 -- 172.21.15.92:0/2239622864 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0031a80 con 0x7f6ae8072380 2024-03-19T03:03:42.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6aee000640 1 -- 172.21.15.92:0/2239622864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8072380 msgr2=0x7f6ae8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6aee000640 1 --2- 172.21.15.92:0/2239622864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8072380 0x7f6ae8071190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009870 tx=0x7f6ad002b900 comp rx=0 tx=0).stop 2024-03-19T03:03:42.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6aee000640 1 -- 172.21.15.92:0/2239622864 shutdown_connections 2024-03-19T03:03:42.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6aee000640 1 --2- 172.21.15.92:0/2239622864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8072380 0x7f6ae8071190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.605+0000 7f6aee000640 1 -- 172.21.15.92:0/2239622864 >> 172.21.15.92:0/2239622864 conn(0x7f6ae80c8d90 msgr2=0x7f6ae80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:42.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.606+0000 7f6aee000640 1 -- 172.21.15.92:0/2239622864 shutdown_connections 2024-03-19T03:03:42.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.606+0000 7f6aee000640 1 -- 172.21.15.92:0/2239622864 wait complete. 2024-03-19T03:03:42.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.607+0000 7f6aee000640 1 Processor -- start 2024-03-19T03:03:42.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.607+0000 7f6aee000640 1 -- start start 2024-03-19T03:03:42.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ae8072380 0x7f6ae815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8154dc0 0x7f6ae8156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80c7520 con 0x7f6ae8154dc0 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80c73a0 con 0x7f6ae80c60d0 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.608+0000 7f6aee000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae80c76a0 con 0x7f6ae8072380 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:41848/0 (socket says 172.21.15.92:41848) 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 -- 172.21.15.92:0/1073975184 learned_addr learned my addr 172.21.15.92:0/1073975184 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae7e00640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8154dc0 0x7f6ae8156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae7400640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ae8072380 0x7f6ae815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 -- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ae8072380 msgr2=0x7f6ae815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ae8072380 0x7f6ae815a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 -- 172.21.15.92:0/1073975184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8154dc0 msgr2=0x7f6ae8156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8154dc0 0x7f6ae8156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.612 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.609+0000 7f6ae6a00640 1 -- 172.21.15.92:0/1073975184 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae81f0a30 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6ae6a00640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0000c00 tx=0x7f6ad00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0040070 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad0039bf0 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0035040 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae81f0d10 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.610+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81f12d0 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.611+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae80c7a60 con 0x7f6ae80c60d0 2024-03-19T03:03:42.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.612+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ad004b060 con 0x7f6ae80c60d0 2024-03-19T03:03:42.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.613+0000 7f6adf400640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 0x7f6ac406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:42.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.613+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f6ad00ac940 con 0x7f6ae80c60d0 2024-03-19T03:03:42.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.613+0000 7f6ae7400640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 0x7f6ac406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:42.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.614+0000 7f6ae7400640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 0x7f6ac406e0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006fd0 tx=0x7f6ad8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:42.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.618+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ae80c7a60 con 0x7f6ae80c60d0 2024-03-19T03:03:42.856 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.855+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ae8153630 con 0x7f6ae80c60d0 2024-03-19T03:03:42.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.856+0000 7f6adf400640 1 -- 172.21.15.92:0/1073975184 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6004 (secure 0 0 0) 0x7f6ad007b1a0 con 0x7f6ae80c60d0 2024-03-19T03:03:42.858 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:42.858 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:38.079258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:42.858 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 28 2024-03-19T03:03:42.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.858+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 msgr2=0x7f6ac406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.858+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 0x7f6ac406e0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006fd0 tx=0x7f6ad8008040 comp rx=0 tx=0).stop 2024-03-19T03:03:42.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.858+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 msgr2=0x7f6ae81567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:42.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.858+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0000c00 tx=0x7f6ad00046c0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.860 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 shutdown_connections 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6ac406bc00 0x7f6ac406e0c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ae8154dc0 0x7f6ae8156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ae80c60d0 0x7f6ae81567c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 --2- 172.21.15.92:0/1073975184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ae8072380 0x7f6ae815a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 >> 172.21.15.92:0/1073975184 conn(0x7f6ae80c8d90 msgr2=0x7f6ae806e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 shutdown_connections 2024-03-19T03:03:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:42.859+0000 7f6aee000640 1 -- 172.21.15.92:0/1073975184 wait complete. 2024-03-19T03:03:42.881 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:44.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:45.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.099+0000 7fa18c000640 1 Processor -- start 2024-03-19T03:03:45.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.099+0000 7fa18c000640 1 -- start start 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.100+0000 7fa18c000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 0x7fa1840c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.100+0000 7fa18c000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa1841593d0 0x7fa1840c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.100+0000 7fa18c000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1840c9a80 con 0x7fa184152bf0 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.100+0000 7fa18c000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fa1840c6c90 con 0x7fa1841593d0 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.100+0000 7fa18c000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa184159910 con 0x7fa1840c6470 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa18a000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa1841593d0 0x7fa1840c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa188c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 0x7fa1840c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa189600640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa184152bf0 0x7fa18406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:52558/0 (socket says 172.21.15.92:52558) 2024-03-19T03:03:45.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa189600640 1 -- 172.21.15.92:0/447957274 learned_addr learned my addr 172.21.15.92:0/447957274 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:45.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa183e00640 1 -- 172.21.15.92:0/447957274 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245933045 0 0) 0x7fa1840c9a80 con 0x7fa184152bf0 2024-03-19T03:03:45.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa188c00640 1 -- 172.21.15.92:0/447957274 >> v2:172.21.15.159:3300/0 conn(0x7fa1841593d0 msgr2=0x7fa1840c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:45.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa188c00640 1 --2- 172.21.15.92:0/447957274 >> v2:172.21.15.159:3300/0 conn(0x7fa1841593d0 0x7fa1840c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa188c00640 1 -- 172.21.15.92:0/447957274 >> v1:172.21.15.159:6789/0 conn(0x7fa184152bf0 legacy=0x7fa18406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.101+0000 7fa188c00640 1 -- 172.21.15.92:0/447957274 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1840c8650 con 0x7fa1840c6470 2024-03-19T03:03:45.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.102+0000 7fa188c00640 1 --2- 172.21.15.92:0/447957274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 0x7fa1840c3f80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1700097a0 tx=0x7fa17002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=166451a3cec6de07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:45.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa183e00640 1 -- 172.21.15.92:0/447957274 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170031780 con 0x7fa1840c6470 2024-03-19T03:03:45.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa183e00640 1 -- 172.21.15.92:0/447957274 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa170037070 con 0x7fa1840c6470 2024-03-19T03:03:45.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa183e00640 1 -- 172.21.15.92:0/447957274 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170031b40 con 0x7fa1840c6470 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 -- 172.21.15.92:0/447957274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 msgr2=0x7fa1840c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 --2- 172.21.15.92:0/447957274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 0x7fa1840c3f80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1700097a0 tx=0x7fa17002b820 comp rx=0 tx=0).stop 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 -- 172.21.15.92:0/447957274 shutdown_connections 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 --2- 172.21.15.92:0/447957274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1840c6470 0x7fa1840c3f80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 --2- 172.21.15.92:0/447957274 >> v2:172.21.15.159:3300/0 conn(0x7fa1841593d0 0x7fa1840c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 -- 172.21.15.92:0/447957274 >> 172.21.15.92:0/447957274 conn(0x7fa1840c8d90 msgr2=0x7fa1840c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 -- 172.21.15.92:0/447957274 shutdown_connections 2024-03-19T03:03:45.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.103+0000 7fa18c000640 1 -- 172.21.15.92:0/447957274 wait complete. 2024-03-19T03:03:45.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.105+0000 7fa18c000640 1 Processor -- start 2024-03-19T03:03:45.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.105+0000 7fa18c000640 1 -- start start 2024-03-19T03:03:45.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 0x7fa1841f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 0x7fa184072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa18406ea60 con 0x7fa1841593d0 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa18406e4a0 con 0x7fa1840c6470 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18c000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa18406e6f0 con 0x7fa184152bf0 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa189600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa18a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 0x7fa184072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa188c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 0x7fa1841f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa189600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:41868/0 (socket says 172.21.15.92:41868) 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.106+0000 7fa189600640 1 -- 172.21.15.92:0/1802099864 learned_addr learned my addr 172.21.15.92:0/1802099864 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 -- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 msgr2=0x7fa1841f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 0x7fa1841f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 -- 172.21.15.92:0/1802099864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 msgr2=0x7fa184072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 0x7fa184072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 -- 172.21.15.92:0/1802099864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1841f4c80 con 0x7fa1840c6470 2024-03-19T03:03:45.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa18a000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 0x7fa184072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:45.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa188c00640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 0x7fa1841f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:45.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa189600640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa17800c500 tx=0x7fa17800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178018070 con 0x7fa1840c6470 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.107+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa178007580 con 0x7fa1840c6470 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.108+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1841e3780 con 0x7fa1840c6470 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.108+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178004a20 con 0x7fa1840c6470 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.108+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1841e3d40 con 0x7fa1840c6470 2024-03-19T03:03:45.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.109+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa184152fd0 con 0x7fa1840c6470 2024-03-19T03:03:45.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.110+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa17801b070 con 0x7fa1840c6470 2024-03-19T03:03:45.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.111+0000 7fa181600640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:45.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.111+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fa178089e70 con 0x7fa1840c6470 2024-03-19T03:03:45.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.112+0000 7fa188c00640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:45.114 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.113+0000 7fa188c00640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 0x7fa16406e0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa170036000 tx=0x7fa170008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:45.118 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.117+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa184152fd0 con 0x7fa1840c6470 2024-03-19T03:03:45.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.352+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1840c2a60 con 0x7fa1840c6470 2024-03-19T03:03:45.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.353+0000 7fa181600640 1 -- 172.21.15.92:0/1802099864 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6010 (secure 0 0 0) 0x7fa1840c2a60 con 0x7fa1840c6470 2024-03-19T03:03:45.355 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:45.355 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:44.198540+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:45.355 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 29 2024-03-19T03:03:45.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 msgr2=0x7fa16406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 0x7fa16406e0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa170036000 tx=0x7fa170008030 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 msgr2=0x7fa18415c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa17800c500 tx=0x7fa17800c9c0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 shutdown_connections 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1841593d0 0x7fa184072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa184152bf0 0x7fa1841f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 --2- 172.21.15.92:0/1802099864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1840c6470 0x7fa18415c670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 >> 172.21.15.92:0/1802099864 conn(0x7fa1840c8d90 msgr2=0x7fa1840c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 shutdown_connections 2024-03-19T03:03:45.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:45.356+0000 7fa18c000640 1 -- 172.21.15.92:0/1802099864 wait complete. 2024-03-19T03:03:45.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:47.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:47.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.596+0000 7ff46a600640 1 Processor -- start 2024-03-19T03:03:47.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.596+0000 7ff46a600640 1 -- start start 2024-03-19T03:03:47.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.597+0000 7ff46a600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 0x7ff464158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.597+0000 7ff46a600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff464151ce0 0x7ff4640723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.597+0000 7ff46a600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4640c3b80 con 0x7ff4640c7da0 2024-03-19T03:03:47.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.597+0000 7ff46a600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff464158860 con 0x7ff464151ce0 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.597+0000 7ff46a600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff464072e70 con 0x7ff464151900 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff463e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff4640c7da0 0x7ff46415adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33820/0 (socket says 172.21.15.92:33820) 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff463e00640 1 -- 172.21.15.92:0/3335986793 learned_addr learned my addr 172.21.15.92:0/3335986793 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff45be00640 1 --2- 172.21.15.92:0/3335986793 >> v2:172.21.15.92:3300/0 conn(0x7ff464151ce0 0x7ff4640723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff463400640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 0x7ff464158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff45be00640 1 -- 172.21.15.92:0/3335986793 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 msgr2=0x7ff464158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:47.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff45be00640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 0x7ff464158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff45be00640 1 -- 172.21.15.92:0/3335986793 >> v1:172.21.15.92:6789/0 conn(0x7ff4640c7da0 legacy=0x7ff46415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff45be00640 1 -- 172.21.15.92:0/3335986793 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff464073150 con 0x7ff464151ce0 2024-03-19T03:03:47.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.598+0000 7ff463400640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 0x7ff464158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.599+0000 7ff45be00640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151ce0 0x7ff4640723d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804a0818778cee12 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff462a00640 1 -- 172.21.15.92:0/3335986793 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c034070 con 0x7ff464151ce0 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff462a00640 1 -- 172.21.15.92:0/3335986793 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c030800 con 0x7ff464151ce0 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff462a00640 1 -- 172.21.15.92:0/3335986793 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c030bd0 con 0x7ff464151ce0 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 -- 172.21.15.92:0/3335986793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151ce0 msgr2=0x7ff4640723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151ce0 0x7ff4640723d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02b940 comp rx=0 tx=0).stop 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 -- 172.21.15.92:0/3335986793 shutdown_connections 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151ce0 0x7ff4640723d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 --2- 172.21.15.92:0/3335986793 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151900 0x7ff464158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 -- 172.21.15.92:0/3335986793 >> 172.21.15.92:0/3335986793 conn(0x7ff4640c8d90 msgr2=0x7ff4640c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:47.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.600+0000 7ff46a600640 1 -- 172.21.15.92:0/3335986793 shutdown_connections 2024-03-19T03:03:47.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.601+0000 7ff46a600640 1 -- 172.21.15.92:0/3335986793 wait complete. 2024-03-19T03:03:47.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.602+0000 7ff46a600640 1 Processor -- start 2024-03-19T03:03:47.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.602+0000 7ff46a600640 1 -- start start 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4640c7da0 0x7ff4641f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151ce0 0x7ff46413ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff464072910 con 0x7ff464151900 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff464071eb0 con 0x7ff4640c7da0 2024-03-19T03:03:47.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff46a600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff4640c9af0 con 0x7ff464151ce0 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff45be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.603+0000 7ff45be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37744/0 (socket says 172.21.15.92:37744) 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 -- 172.21.15.92:0/1113036568 learned_addr learned my addr 172.21.15.92:0/1113036568 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff463400640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4640c7da0 0x7ff4641f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff463e00640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151ce0 0x7ff46413ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 -- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151ce0 msgr2=0x7ff46413ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151ce0 0x7ff46413ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 -- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4640c7da0 msgr2=0x7ff4641f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4640c7da0 0x7ff4641f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 -- 172.21.15.92:0/1113036568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4641545d0 con 0x7ff464151900 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45be00640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff44c02bf50 tx=0x7ff44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.604+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c034040 con 0x7ff464151900 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.605+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c030600 con 0x7ff464151900 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.605+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff464154870 con 0x7ff464151900 2024-03-19T03:03:47.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.605+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0423f0 con 0x7ff464151900 2024-03-19T03:03:47.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.605+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff464154e30 con 0x7ff464151900 2024-03-19T03:03:47.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.606+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4640684c0 con 0x7ff464151900 2024-03-19T03:03:47.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.607+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff44c037070 con 0x7ff464151900 2024-03-19T03:03:47.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.607+0000 7ff45b400640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 0x7ff44006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:47.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.608+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7ff44c0acca0 con 0x7ff464151900 2024-03-19T03:03:47.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.608+0000 7ff463400640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 0x7ff44006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:47.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.609+0000 7ff463400640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 0x7ff44006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff454006fd0 tx=0x7ff454008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:47.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.612+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff44c07af40 con 0x7ff464151900 2024-03-19T03:03:47.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.847+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4640c4b80 con 0x7ff464151900 2024-03-19T03:03:47.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.848+0000 7ff45b400640 1 -- 172.21.15.92:0/1113036568 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6010 (secure 0 0 0) 0x7ff44c07efe0 con 0x7ff464151900 2024-03-19T03:03:47.850 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:47.850 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:44.198540+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:47.850 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 29 2024-03-19T03:03:47.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.850+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 msgr2=0x7ff44006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.850+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 0x7ff44006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff454006fd0 tx=0x7ff454008040 comp rx=0 tx=0).stop 2024-03-19T03:03:47.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.850+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 msgr2=0x7ff464155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:47.852 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.850+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff44c02bf50 tx=0x7ff44c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 shutdown_connections 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff44006bbd0 0x7ff44006e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff464151ce0 0x7ff46413ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464151900 0x7ff464155da0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 --2- 172.21.15.92:0/1113036568 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff4640c7da0 0x7ff4641f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 >> 172.21.15.92:0/1113036568 conn(0x7ff4640c8d90 msgr2=0x7ff4640cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 shutdown_connections 2024-03-19T03:03:47.853 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:47.851+0000 7ff46a600640 1 -- 172.21.15.92:0/1113036568 wait complete. 2024-03-19T03:03:47.872 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:49.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:50.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.092+0000 7fd3dd600640 1 Processor -- start 2024-03-19T03:03:50.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.093+0000 7fd3dd600640 1 -- start start 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.094+0000 7fd3dd600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 0x7fd3d8071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.094+0000 7fd3dd600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd3d8072380 0x7fd3d80c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.094+0000 7fd3dd600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d815b740 con 0x7fd3d80c60d0 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.094+0000 7fd3dd600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d815ba20 con 0x7fd3d8072380 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.094+0000 7fd3dd600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d80c8400 con 0x7fd3d8154dc0 2024-03-19T03:03:50.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3ce000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd3d80c60d0 0x7fd3d8071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:33828/0 (socket says 172.21.15.92:33828) 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3ce000640 1 -- 172.21.15.92:0/43973025 learned_addr learned my addr 172.21.15.92:0/43973025 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/43973025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 0x7fd3d8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d7400640 1 --2- 172.21.15.92:0/43973025 >> v2:172.21.15.159:3300/0 conn(0x7fd3d8072380 0x7fd3d80c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6a00640 1 -- 172.21.15.92:0/43973025 >> v2:172.21.15.159:3300/0 conn(0x7fd3d8072380 msgr2=0x7fd3d80c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6000640 1 -- 172.21.15.92:0/43973025 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889642675 0 0) 0x7fd3d815b740 con 0x7fd3d80c60d0 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/43973025 >> v2:172.21.15.159:3300/0 conn(0x7fd3d8072380 0x7fd3d80c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6a00640 1 -- 172.21.15.92:0/43973025 >> v1:172.21.15.92:6789/0 conn(0x7fd3d80c60d0 legacy=0x7fd3d8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.095+0000 7fd3d6a00640 1 -- 172.21.15.92:0/43973025 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d80c86e0 con 0x7fd3d8154dc0 2024-03-19T03:03:50.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.097+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/43973025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 0x7fd3d8071190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097a0 tx=0x7fd3c802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7da323aa4c6a57e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:50.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.097+0000 7fd3d6000640 1 -- 172.21.15.92:0/43973025 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8031780 con 0x7fd3d8154dc0 2024-03-19T03:03:50.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.097+0000 7fd3d6000640 1 -- 172.21.15.92:0/43973025 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c8037070 con 0x7fd3d8154dc0 2024-03-19T03:03:50.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.097+0000 7fd3d6000640 1 -- 172.21.15.92:0/43973025 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8031b40 con 0x7fd3d8154dc0 2024-03-19T03:03:50.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 -- 172.21.15.92:0/43973025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 msgr2=0x7fd3d8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 --2- 172.21.15.92:0/43973025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 0x7fd3d8071190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097a0 tx=0x7fd3c802b820 comp rx=0 tx=0).stop 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 -- 172.21.15.92:0/43973025 shutdown_connections 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 --2- 172.21.15.92:0/43973025 >> v2:172.21.15.159:3300/0 conn(0x7fd3d8072380 0x7fd3d80c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 --2- 172.21.15.92:0/43973025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8154dc0 0x7fd3d8071190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 -- 172.21.15.92:0/43973025 >> 172.21.15.92:0/43973025 conn(0x7fd3d80c8d90 msgr2=0x7fd3d80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 -- 172.21.15.92:0/43973025 shutdown_connections 2024-03-19T03:03:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.098+0000 7fd3dd600640 1 -- 172.21.15.92:0/43973025 wait complete. 2024-03-19T03:03:50.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.100+0000 7fd3dd600640 1 Processor -- start 2024-03-19T03:03:50.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.100+0000 7fd3dd600640 1 -- start start 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8072380 0x7fd3d815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3d80c60d0 0x7fd3d815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d80c7350 con 0x7fd3d8154dc0 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d80c71d0 con 0x7fd3d80c60d0 2024-03-19T03:03:50.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3dd600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d80c74d0 con 0x7fd3d8072380 2024-03-19T03:03:50.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37756/0 (socket says 172.21.15.92:37756) 2024-03-19T03:03:50.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3d7400640 1 -- 172.21.15.92:0/2602330586 learned_addr learned my addr 172.21.15.92:0/2602330586 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:50.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.101+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8072380 0x7fd3d815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3ce000640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3d80c60d0 0x7fd3d815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 -- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8072380 msgr2=0x7fd3d815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8072380 0x7fd3d815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 -- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3d80c60d0 msgr2=0x7fd3d815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3d80c60d0 0x7fd3d815dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 -- 172.21.15.92:0/2602330586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d815e610 con 0x7fd3d8154dc0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3d7400640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000efc0 tx=0x7fd3d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.102+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0018020 con 0x7fd3d8154dc0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.103+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d0014480 con 0x7fd3d8154dc0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.103+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0014870 con 0x7fd3d8154dc0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.103+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d815bea0 con 0x7fd3d8154dc0 2024-03-19T03:03:50.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.103+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d81f4500 con 0x7fd3d8154dc0 2024-03-19T03:03:50.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.104+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d80684c0 con 0x7fd3d8154dc0 2024-03-19T03:03:50.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.105+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd3d0005d10 con 0x7fd3d8154dc0 2024-03-19T03:03:50.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.106+0000 7fd3cf400640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:50.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.106+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fd3d008b940 con 0x7fd3d8154dc0 2024-03-19T03:03:50.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.106+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:50.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.107+0000 7fd3d6a00640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 0x7fd3b406e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8036000 tx=0x7fd3c803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:50.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.111+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd3d0022030 con 0x7fd3d8154dc0 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.380+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3d8146a50 con 0x7fd3d8154dc0 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.381+0000 7fd3cf400640 1 -- 172.21.15.92:0/2602330586 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6010 (secure 0 0 0) 0x7fd3d815e610 con 0x7fd3d8154dc0 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:44.198540+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 29 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 msgr2=0x7fd3b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 0x7fd3b406e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8036000 tx=0x7fd3c803d040 comp rx=0 tx=0).stop 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 msgr2=0x7fd3d8142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000efc0 tx=0x7fd3d000c490 comp rx=0 tx=0).stop 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 shutdown_connections 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.384+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d8154dc0 0x7fd3d8142f40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.385+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3d80c60d0 0x7fd3d815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.385+0000 7fd3dd600640 1 --2- 172.21.15.92:0/2602330586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3d8072380 0x7fd3d815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.385+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 >> 172.21.15.92:0/2602330586 conn(0x7fd3d80c8d90 msgr2=0x7fd3d80c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.385+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 shutdown_connections 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:50.385+0000 7fd3dd600640 1 -- 172.21.15.92:0/2602330586 wait complete. 2024-03-19T03:03:50.405 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5185": { "addr": "172.21.15.159:6839/3845908330", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6838", "nonce": 3845908330, "type": "v2" }, { "addr": "172.21.15.159:6839", "nonce": 3845908330, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5185, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:03:44.198540+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5185 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:03:52.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:52.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.617+0000 7fbefb200640 1 Processor -- start 2024-03-19T03:03:52.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.617+0000 7fbefb200640 1 -- start start 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefb200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 0x7fbefc152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefb200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbefc06d160 0x7fbefc152a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefb200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbefc152fa0 con 0x7fbefc0c5b70 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefb200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fbefc15c580 con 0x7fbefc06d160 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefb200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbefc06df20 con 0x7fbefc0c5f50 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbef9e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbefc0c5b70 0x7fbefc15be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:60578/0 (socket says 172.21.15.92:60578) 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbefa800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbefc06d160 0x7fbefc152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbef9400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 0x7fbefc152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.619+0000 7fbef9e00640 1 -- 172.21.15.92:0/2149956023 learned_addr learned my addr 172.21.15.92:0/2149956023 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:03:52.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbefa800640 1 -- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 msgr2=0x7fbefc152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:52.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbefa800640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 0x7fbefc152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbefa800640 1 -- 172.21.15.92:0/2149956023 >> v1:172.21.15.159:6789/0 conn(0x7fbefc0c5b70 legacy=0x7fbefc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbefa800640 1 -- 172.21.15.92:0/2149956023 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc0c6d00 con 0x7fbefc06d160 2024-03-19T03:03:52.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbef8a00640 1 -- 172.21.15.92:0/2149956023 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541214440 0 0) 0x7fbefc152fa0 con 0x7fbefc0c5b70 2024-03-19T03:03:52.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.620+0000 7fbef9400640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 0x7fbefc152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:52.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.621+0000 7fbefa800640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc152a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009d90 tx=0x7fbef402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acfb1a347f012f4d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:52.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.621+0000 7fbef8a00640 1 -- 172.21.15.92:0/2149956023 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4039070 con 0x7fbefc06d160 2024-03-19T03:03:52.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.621+0000 7fbef8a00640 1 -- 172.21.15.92:0/2149956023 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef402cce0 con 0x7fbefc06d160 2024-03-19T03:03:52.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.621+0000 7fbef8a00640 1 -- 172.21.15.92:0/2149956023 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4034660 con 0x7fbefc06d160 2024-03-19T03:03:52.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 -- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 msgr2=0x7fbefc152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc152a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009d90 tx=0x7fbef402bf30 comp rx=0 tx=0).stop 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 -- 172.21.15.92:0/2149956023 shutdown_connections 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5f50 0x7fbefc152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 --2- 172.21.15.92:0/2149956023 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc152a60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 -- 172.21.15.92:0/2149956023 >> 172.21.15.92:0/2149956023 conn(0x7fbefc0cc1f0 msgr2=0x7fbefc0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 -- 172.21.15.92:0/2149956023 shutdown_connections 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.622+0000 7fbefb200640 1 -- 172.21.15.92:0/2149956023 wait complete. 2024-03-19T03:03:52.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.624+0000 7fbefb200640 1 Processor -- start 2024-03-19T03:03:52.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.624+0000 7fbefb200640 1 -- start start 2024-03-19T03:03:52.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc0c5f50 0x7fbefc1f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc06d8d0 con 0x7fbefc0c5f50 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc0c6fa0 con 0x7fbefc06d160 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.625+0000 7fbefb200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbefc06dbb0 con 0x7fbefc0c5b70 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42710/0 (socket says 172.21.15.92:42710) 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9400640 1 -- 172.21.15.92:0/327295246 learned_addr learned my addr 172.21.15.92:0/327295246 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbefa800640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc0c5f50 0x7fbefc1f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 -- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 msgr2=0x7fbefc1f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 -- 172.21.15.92:0/327295246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc0c5f50 msgr2=0x7fbefc1f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc0c5f50 0x7fbefc1f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9e00640 1 -- 172.21.15.92:0/327295246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc15a6a0 con 0x7fbefc06d160 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.626+0000 7fbef9400640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbef9e00640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc15bf60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c480 tx=0x7fbeec00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec018070 con 0x7fbefc06d160 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec0074e0 con 0x7fbefc06d160 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc15a980 con 0x7fbefc06d160 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004a20 con 0x7fbefc06d160 2024-03-19T03:03:52.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.627+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbefc15af40 con 0x7fbefc06d160 2024-03-19T03:03:52.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.628+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbefc04f580 con 0x7fbefc06d160 2024-03-19T03:03:52.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.629+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbeec01b070 con 0x7fbefc06d160 2024-03-19T03:03:52.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.630+0000 7fbef2000640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:52.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.630+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fbeec089f00 con 0x7fbefc06d160 2024-03-19T03:03:52.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.630+0000 7fbef9400640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:52.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.631+0000 7fbef9400640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbefc1ea740 tx=0x7fbee8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:52.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.635+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbeec0581a0 con 0x7fbefc06d160 2024-03-19T03:03:52.886 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.885+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbefc06e620 con 0x7fbefc06d160 2024-03-19T03:03:52.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.887+0000 7fbef2000640 1 -- 172.21.15.92:0/327295246 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6010 (secure 0 0 0) 0x7fbeec05c240 con 0x7fbefc06d160 2024-03-19T03:03:52.887 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:52.888 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:44.198540+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:03:52.888 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 29 2024-03-19T03:03:52.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.888+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 msgr2=0x7fbedc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.888+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbefc1ea740 tx=0x7fbee8009190 comp rx=0 tx=0).stop 2024-03-19T03:03:52.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.888+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 msgr2=0x7fbefc15bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:52.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.888+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc15bf60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c480 tx=0x7fbeec00c940 comp rx=0 tx=0).stop 2024-03-19T03:03:52.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 shutdown_connections 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc0c5f50 0x7fbefc1f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbefc0c5b70 0x7fbefc1f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 --2- 172.21.15.92:0/327295246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbefc06d160 0x7fbefc15bf60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 >> 172.21.15.92:0/327295246 conn(0x7fbefc0cc1f0 msgr2=0x7fbefc0cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 shutdown_connections 2024-03-19T03:03:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:52.889+0000 7fbefb200640 1 -- 172.21.15.92:0/327295246 wait complete. 2024-03-19T03:03:52.909 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:03:54.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:55.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.127+0000 7fa3dee00640 1 Processor -- start 2024-03-19T03:03:55.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.128+0000 7fa3dee00640 1 -- start start 2024-03-19T03:03:55.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dee00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3d80c5b70 0x7fa3d815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d8152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dee00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d80c6d00 con 0x7fa3d806d160 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dee00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d806df20 con 0x7fa3d80c5b70 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d806d610 con 0x7fa3d80c5f50 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dda00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3d80c5b70 0x7fa3d815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dda00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3d80c5b70 0x7fa3d815be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37768/0 (socket says 172.21.15.92:37768) 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.129+0000 7fa3dda00640 1 -- 172.21.15.92:0/1014391286 learned_addr learned my addr 172.21.15.92:0/1014391286 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.130+0000 7fa3dd000640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.130+0000 7fa3dda00640 1 -- 172.21.15.92:0/1014391286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 msgr2=0x7fa3d8152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.130+0000 7fa3dda00640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d8152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.130+0000 7fa3dda00640 1 -- 172.21.15.92:0/1014391286 >> v1:172.21.15.159:6789/0 conn(0x7fa3d806d160 legacy=0x7fa3d8152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.130+0000 7fa3dda00640 1 -- 172.21.15.92:0/1014391286 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d80c6fa0 con 0x7fa3d80c5b70 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.131+0000 7fa3dda00640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80c5b70 0x7fa3d815be70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009870 tx=0x7fa3cc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede1de82d82b2103 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.131+0000 7fa3d7e00640 1 -- 172.21.15.92:0/1014391286 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc02f840 con 0x7fa3d80c5b70 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.131+0000 7fa3d7e00640 1 -- 172.21.15.92:0/1014391286 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3cc02f9e0 con 0x7fa3d80c5b70 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.131+0000 7fa3d7e00640 1 -- 172.21.15.92:0/1014391286 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc02fdb0 con 0x7fa3d80c5b70 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 -- 172.21.15.92:0/1014391286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80c5b70 msgr2=0x7fa3d815be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80c5b70 0x7fa3d815be70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009870 tx=0x7fa3cc02ba20 comp rx=0 tx=0).stop 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 -- 172.21.15.92:0/1014391286 shutdown_connections 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d8152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 --2- 172.21.15.92:0/1014391286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80c5b70 0x7fa3d815be70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 -- 172.21.15.92:0/1014391286 >> 172.21.15.92:0/1014391286 conn(0x7fa3d80cc1f0 msgr2=0x7fa3d80cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 -- 172.21.15.92:0/1014391286 shutdown_connections 2024-03-19T03:03:55.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.132+0000 7fa3dee00640 1 -- 172.21.15.92:0/1014391286 wait complete. 2024-03-19T03:03:55.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.134+0000 7fa3dee00640 1 Processor -- start 2024-03-19T03:03:55.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.134+0000 7fa3dee00640 1 -- start start 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 0x7fa3d8149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d81f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d80729b0 con 0x7fa3d806d160 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d806dbb0 con 0x7fa3d80c5b70 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d806e3b0 con 0x7fa3d80c5f50 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:37774/0 (socket says 172.21.15.92:37774) 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dda00640 1 -- 172.21.15.92:0/480388346 learned_addr learned my addr 172.21.15.92:0/480388346 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3dd000640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 0x7fa3d8149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.135+0000 7fa3de400640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d81f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 -- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 msgr2=0x7fa3d81f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d81f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 -- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 msgr2=0x7fa3d8149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 0x7fa3d8149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 -- 172.21.15.92:0/480388346 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d81f4de0 con 0x7fa3d806d160 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dd000640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 0x7fa3d8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dda00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc02ba60 tx=0x7fa3cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc047070 con 0x7fa3d806d160 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3cc037bc0 con 0x7fa3d806d160 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d81f0910 con 0x7fa3d806d160 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc042480 con 0x7fa3d806d160 2024-03-19T03:03:55.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.136+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d81f0b90 con 0x7fa3d806d160 2024-03-19T03:03:55.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.138+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d804f580 con 0x7fa3d806d160 2024-03-19T03:03:55.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.138+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3cc04a070 con 0x7fa3d806d160 2024-03-19T03:03:55.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.139+0000 7fa3d5600640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 0x7fa3b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:55.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.139+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fa3cc0acf30 con 0x7fa3d806d160 2024-03-19T03:03:55.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.140+0000 7fa3dd000640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 0x7fa3b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:55.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.140+0000 7fa3dd000640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 0x7fa3b806e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8158da0 tx=0x7fa3c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:55.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.144+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3d81f0910 con 0x7fa3d806d160 2024-03-19T03:03:55.384 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.383+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3d815a420 con 0x7fa3d806d160 2024-03-19T03:03:55.385 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.385+0000 7fa3d5600640 1 -- 172.21.15.92:0/480388346 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6011 (secure 0 0 0) 0x7fa3cc033030 con 0x7fa3d806d160 2024-03-19T03:03:55.385 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:03:55.386 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:03:55.386 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 30 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 msgr2=0x7fa3b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 0x7fa3b806e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8158da0 tx=0x7fa3c4008040 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 msgr2=0x7fa3d81f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc02ba60 tx=0x7fa3cc004730 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 shutdown_connections 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa3b806bbd0 0x7fa3b806e090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80c5f50 0x7fa3d81f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80c5b70 0x7fa3d8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 --2- 172.21.15.92:0/480388346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d806d160 0x7fa3d81f48a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 >> 172.21.15.92:0/480388346 conn(0x7fa3d80cc1f0 msgr2=0x7fa3d80cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 shutdown_connections 2024-03-19T03:03:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:03:55.388+0000 7fa3dee00640 1 -- 172.21.15.92:0/480388346 wait complete. 2024-03-19T03:03:55.408 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5041 2024-03-19T03:03:55.409 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2024-03-19T03:04:01.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:04:01.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:04:02.409 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-19T03:04:02.410 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-19T03:04:02.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-19T03:04:02.412 INFO:tasks.ceph.mds.f:Started 2024-03-19T03:04:02.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:02.487 INFO:tasks.ceph.mds.f.smithi159.stdout:starting mds.f at 2024-03-19T03:04:02.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.632+0000 7f8386800640 1 Processor -- start 2024-03-19T03:04:02.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.632+0000 7f8386800640 1 -- start start 2024-03-19T03:04:02.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f8386800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 0x7f83800c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f8386800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 0x7f8380159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f8386800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8380072890 con 0x7f838006f800 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f8386800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f8380159780 con 0x7f8380158db0 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f8386800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83800c6c90 con 0x7f8380071da0 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f838006f800 0x7f8380072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58186/0 (socket says 172.21.15.92:58186) 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837ea00640 1 -- 172.21.15.92:0/3148534251 learned_addr learned my addr 172.21.15.92:0/3148534251 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837f400640 1 --2- 172.21.15.92:0/3148534251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 0x7f83800c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837fe00640 1 --2- 172.21.15.92:0/3148534251 >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 0x7f8380159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837f400640 1 -- 172.21.15.92:0/3148534251 >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 msgr2=0x7f8380159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:04:02.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837f400640 1 --2- 172.21.15.92:0/3148534251 >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 0x7f8380159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837f400640 1 -- 172.21.15.92:0/3148534251 >> v1:172.21.15.92:6789/0 conn(0x7f838006f800 legacy=0x7f8380072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837f400640 1 -- 172.21.15.92:0/3148534251 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83800c8200 con 0x7f8380071da0 2024-03-19T03:04:02.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837fe00640 1 --2- 172.21.15.92:0/3148534251 >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 0x7f8380159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:02.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.634+0000 7f837e000640 1 -- 172.21.15.92:0/3148534251 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672293952 0 0) 0x7f8380072890 con 0x7f838006f800 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f837f400640 1 --2- 172.21.15.92:0/3148534251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 0x7f83800c8740 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83700097a0 tx=0x7f837002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d9be94a17745820 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f837e000640 1 -- 172.21.15.92:0/3148534251 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370035020 con 0x7f8380071da0 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f837e000640 1 -- 172.21.15.92:0/3148534251 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8370031830 con 0x7f8380071da0 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f837e000640 1 -- 172.21.15.92:0/3148534251 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370031c00 con 0x7f8380071da0 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 -- 172.21.15.92:0/3148534251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 msgr2=0x7f83800c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 --2- 172.21.15.92:0/3148534251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 0x7f83800c8740 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83700097a0 tx=0x7f837002b8d0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 -- 172.21.15.92:0/3148534251 shutdown_connections 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 --2- 172.21.15.92:0/3148534251 >> v2:172.21.15.159:3300/0 conn(0x7f8380158db0 0x7f8380159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 --2- 172.21.15.92:0/3148534251 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380071da0 0x7f83800c8740 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 -- 172.21.15.92:0/3148534251 >> 172.21.15.92:0/3148534251 conn(0x7f83800c8d90 msgr2=0x7f83800c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:02.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.636+0000 7f8386800640 1 -- 172.21.15.92:0/3148534251 shutdown_connections 2024-03-19T03:04:02.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.637+0000 7f8386800640 1 -- 172.21.15.92:0/3148534251 wait complete. 2024-03-19T03:04:02.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.638+0000 7f8386800640 1 Processor -- start 2024-03-19T03:04:02.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.638+0000 7f8386800640 1 -- start start 2024-03-19T03:04:02.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 0x7f8380159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8380071da0 0x7f838015de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f838015fde0 con 0x7f8380071da0 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f838015fb00 con 0x7f838006f800 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f8386800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83800c7120 con 0x7f8380158db0 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f837f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 0x7f8380159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f837fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.639+0000 7f837ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8380071da0 0x7f838015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:58830/0 (socket says 172.21.15.92:58830) 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 0x7f8380159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:53772/0 (socket says 172.21.15.92:53772) 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 -- 172.21.15.92:0/1792875656 learned_addr learned my addr 172.21.15.92:0/1792875656 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 -- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 msgr2=0x7f8380159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 0x7f8380159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 -- 172.21.15.92:0/1792875656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8380071da0 msgr2=0x7f838015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8380071da0 0x7f838015de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 -- 172.21.15.92:0/1792875656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83801f7a60 con 0x7f8380158db0 2024-03-19T03:04:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f837fe00640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8378009e60 tx=0x7f837800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:02.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378018030 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.640+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f837800cdf0 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.641+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378003bb0 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.641+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83801f7d40 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.641+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83801f8300 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.642+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83800684c0 con 0x7f8380158db0 2024-03-19T03:04:02.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.643+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f837801b070 con 0x7f8380158db0 2024-03-19T03:04:02.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.644+0000 7f8377400640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:02.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.644+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f837808b4d0 con 0x7f8380158db0 2024-03-19T03:04:02.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.644+0000 7f837f400640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:02.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.644+0000 7f837f400640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 0x7f836006e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8370036000 tx=0x7f837003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:02.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.648+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8378010040 con 0x7f8380158db0 2024-03-19T03:04:02.887 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.885+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83800c92a0 con 0x7f8380158db0 2024-03-19T03:04:02.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.886+0000 7f8377400640 1 -- 172.21.15.92:0/1792875656 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6011 (secure 0 0 0) 0x7f8378059720 con 0x7f8380158db0 2024-03-19T03:04:02.888 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:04:02.888 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:04:02.888 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 30 2024-03-19T03:04:02.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.888+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 msgr2=0x7f836006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 0x7f836006e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8370036000 tx=0x7f837003d040 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 msgr2=0x7f838015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8378009e60 tx=0x7f837800c490 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 shutdown_connections 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8380158db0 0x7f838015a280 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8380071da0 0x7f838015de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 --2- 172.21.15.92:0/1792875656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f838006f800 0x7f8380159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 >> 172.21.15.92:0/1792875656 conn(0x7f83800c8d90 msgr2=0x7f83801574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 shutdown_connections 2024-03-19T03:04:02.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:02.889+0000 7f8386800640 1 -- 172.21.15.92:0/1792875656 wait complete. 2024-03-19T03:04:02.911 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-03-19T03:04:03.018 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-03-19T03:04:03.017+0000 7fdf91c8a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T03:04:04.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:05.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.127+0000 7f00b9800640 1 Processor -- start 2024-03-19T03:04:05.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.127+0000 7f00b9800640 1 -- start start 2024-03-19T03:04:05.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.128+0000 7f00b9800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00b4151900 0x7f00b4158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.128+0000 7f00b9800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f00b4151ce0 0x7f00b40723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b9800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00b40c3b80 con 0x7f00b40c7da0 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b9800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f00b4158860 con 0x7f00b4151900 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b9800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f00b4072e70 con 0x7f00b4151ce0 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00aa000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f00b4151ce0 0x7f00b40723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b2a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00b4151900 0x7f00b4158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b3400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f00b40c7da0 0x7f00b415adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58194/0 (socket says 172.21.15.92:58194) 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00b3400640 1 -- 172.21.15.92:0/1318531078 learned_addr learned my addr 172.21.15.92:0/1318531078 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00aa000640 1 -- 172.21.15.92:0/1318531078 >> v2:172.21.15.92:3300/0 conn(0x7f00b4151900 msgr2=0x7f00b4158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00aa000640 1 --2- 172.21.15.92:0/1318531078 >> v2:172.21.15.92:3300/0 conn(0x7f00b4151900 0x7f00b4158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00aa000640 1 -- 172.21.15.92:0/1318531078 >> v1:172.21.15.92:6789/0 conn(0x7f00b40c7da0 legacy=0x7f00b415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.129+0000 7f00aa000640 1 -- 172.21.15.92:0/1318531078 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b4073150 con 0x7f00b4151ce0 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.130+0000 7f00aa000640 1 --2- 172.21.15.92:0/1318531078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b40723d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f00b40c4a30 tx=0x7f009c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec1b635b43c60da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b2000640 1 -- 172.21.15.92:0/1318531078 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c034070 con 0x7f00b4151ce0 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b2000640 1 -- 172.21.15.92:0/1318531078 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c030610 con 0x7f00b4151ce0 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b2000640 1 -- 172.21.15.92:0/1318531078 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0309e0 con 0x7f00b4151ce0 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 -- 172.21.15.92:0/1318531078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 msgr2=0x7f00b40723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 --2- 172.21.15.92:0/1318531078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b40723d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f00b40c4a30 tx=0x7f009c02b960 comp rx=0 tx=0).stop 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 -- 172.21.15.92:0/1318531078 shutdown_connections 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 --2- 172.21.15.92:0/1318531078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b40723d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 --2- 172.21.15.92:0/1318531078 >> v2:172.21.15.92:3300/0 conn(0x7f00b4151900 0x7f00b4158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 -- 172.21.15.92:0/1318531078 >> 172.21.15.92:0/1318531078 conn(0x7f00b40c8d90 msgr2=0x7f00b40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:05.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 -- 172.21.15.92:0/1318531078 shutdown_connections 2024-03-19T03:04:05.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.131+0000 7f00b9800640 1 -- 172.21.15.92:0/1318531078 wait complete. 2024-03-19T03:04:05.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.133+0000 7f00b9800640 1 Processor -- start 2024-03-19T03:04:05.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.133+0000 7f00b9800640 1 -- start start 2024-03-19T03:04:05.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00b4151900 0x7f00b4157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b4147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f00b4072910 con 0x7f00b4151900 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f00b4071eb0 con 0x7f00b4151ce0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b9800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f00b40c9af0 con 0x7f00b40c7da0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b3400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b4147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b2a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b2a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:58840/0 (socket says 172.21.15.92:58840) 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00b2a00640 1 -- 172.21.15.92:0/940340099 learned_addr learned my addr 172.21.15.92:0/940340099 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.134+0000 7f00aa000640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00b4151900 0x7f00b4157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 -- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 msgr2=0x7f00b4147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b4147860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 -- 172.21.15.92:0/940340099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00b4151900 msgr2=0x7f00b4157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00b4151900 0x7f00b4157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 -- 172.21.15.92:0/940340099 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b41534c0 con 0x7f00b40c7da0 2024-03-19T03:04:05.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b2a00640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00a400e9e0 tx=0x7f00a400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:05.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4017070 con 0x7f00b40c7da0 2024-03-19T03:04:05.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a400c9d0 con 0x7f00b40c7da0 2024-03-19T03:04:05.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a400cdc0 con 0x7f00b40c7da0 2024-03-19T03:04:05.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.135+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b41562c0 con 0x7f00b40c7da0 2024-03-19T03:04:05.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.136+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b4156880 con 0x7f00b40c7da0 2024-03-19T03:04:05.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.137+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b40684c0 con 0x7f00b40c7da0 2024-03-19T03:04:05.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.137+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f00a401a070 con 0x7f00b40c7da0 2024-03-19T03:04:05.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.138+0000 7f00ab400640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 0x7f009006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:05.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.139+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f00a408ad70 con 0x7f00b40c7da0 2024-03-19T03:04:05.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.139+0000 7f00aa000640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 0x7f009006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:05.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.139+0000 7f00aa000640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 0x7f009006e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f009c036000 tx=0x7f009c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:05.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.147+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f00a4058ef0 con 0x7f00b40c7da0 2024-03-19T03:04:05.389 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.387+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00b40c54c0 con 0x7f00b40c7da0 2024-03-19T03:04:05.390 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.388+0000 7f00ab400640 1 -- 172.21.15.92:0/940340099 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f00a405cf90 con 0x7f00b40c7da0 2024-03-19T03:04:05.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:04:05.390 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:04:05.391 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 msgr2=0x7f009006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 0x7f009006e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f009c036000 tx=0x7f009c03b040 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 msgr2=0x7f00b406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00a400e9e0 tx=0x7f00a400eea0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 shutdown_connections 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f009006bbd0 0x7f009006e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b4151ce0 0x7f00b4147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00b4151900 0x7f00b4157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 --2- 172.21.15.92:0/940340099 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00b40c7da0 0x7f00b406f160 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:05.393 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 >> 172.21.15.92:0/940340099 conn(0x7f00b40c8d90 msgr2=0x7f00b4150340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:05.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 shutdown_connections 2024-03-19T03:04:05.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:05.391+0000 7f00b9800640 1 -- 172.21.15.92:0/940340099 wait complete. 2024-03-19T03:04:05.413 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-19T03:04:05.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2024-03-19T03:04:31.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:04:31.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:04:55.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:55.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.630+0000 7fc750400640 1 Processor -- start 2024-03-19T03:04:55.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.630+0000 7fc750400640 1 -- start start 2024-03-19T03:04:55.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.631+0000 7fc750400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 0x7fc7480c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:55.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.631+0000 7fc750400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7480c9a80 con 0x7fc7480c6470 2024-03-19T03:04:55.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.631+0000 7fc750400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7480c8650 con 0x7fc748152bf0 2024-03-19T03:04:55.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.631+0000 7fc750400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc7480c6c90 con 0x7fc7481593d0 2024-03-19T03:04:55.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74d000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc7480c6470 0x7fc7480c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36502/0 (socket says 172.21.15.92:36502) 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74d000640 1 -- 172.21.15.92:0/1430395444 learned_addr learned my addr 172.21.15.92:0/1430395444 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74da00640 1 --1- 172.21.15.92:0/1430395444 >> v1:172.21.15.159:6789/0 conn(0x7fc748152bf0 0x7fc74806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:56524/0 (socket says 172.21.15.92:56524) 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74e400640 1 --2- 172.21.15.92:0/1430395444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 0x7fc7480c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc747e00640 1 -- 172.21.15.92:0/1430395444 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1766880800 0 0) 0x7fc7480c8650 con 0x7fc748152bf0 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74e400640 1 -- 172.21.15.92:0/1430395444 >> v1:172.21.15.159:6789/0 conn(0x7fc748152bf0 legacy=0x7fc74806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74e400640 1 -- 172.21.15.92:0/1430395444 >> v1:172.21.15.92:6789/0 conn(0x7fc7480c6470 legacy=0x7fc7480c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.632+0000 7fc74e400640 1 -- 172.21.15.92:0/1430395444 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7480c8930 con 0x7fc7481593d0 2024-03-19T03:04:55.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.633+0000 7fc74e400640 1 --2- 172.21.15.92:0/1430395444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 0x7fc7480c4690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7400097d0 tx=0x7fc74002b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ce4ceb97010f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:55.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc747e00640 1 -- 172.21.15.92:0/1430395444 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740037070 con 0x7fc7481593d0 2024-03-19T03:04:55.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc747e00640 1 -- 172.21.15.92:0/1430395444 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc740030710 con 0x7fc7481593d0 2024-03-19T03:04:55.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc747e00640 1 -- 172.21.15.92:0/1430395444 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740030ae0 con 0x7fc7481593d0 2024-03-19T03:04:55.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 -- 172.21.15.92:0/1430395444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 msgr2=0x7fc7480c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 --2- 172.21.15.92:0/1430395444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 0x7fc7480c4690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7400097d0 tx=0x7fc74002b6e0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 -- 172.21.15.92:0/1430395444 shutdown_connections 2024-03-19T03:04:55.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 --2- 172.21.15.92:0/1430395444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7481593d0 0x7fc7480c4690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 -- 172.21.15.92:0/1430395444 >> 172.21.15.92:0/1430395444 conn(0x7fc7480c8d90 msgr2=0x7fc7480c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:55.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.634+0000 7fc750400640 1 -- 172.21.15.92:0/1430395444 shutdown_connections 2024-03-19T03:04:55.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.635+0000 7fc750400640 1 -- 172.21.15.92:0/1430395444 wait complete. 2024-03-19T03:04:55.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.636+0000 7fc750400640 1 Processor -- start 2024-03-19T03:04:55.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.637+0000 7fc750400640 1 -- start start 2024-03-19T03:04:55.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 0x7fc748071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc74806e4a0 con 0x7fc748152bf0 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc748159910 con 0x7fc7481593d0 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc750400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc74806ea60 con 0x7fc7480c6470 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 0x7fc748071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74e400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74e400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:54494/0 (socket says 172.21.15.92:54494) 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52424/0 (socket says 172.21.15.92:52424) 2024-03-19T03:04:55.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74e400640 1 -- 172.21.15.92:0/1661882594 learned_addr learned my addr 172.21.15.92:0/1661882594 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:04:55.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74e400640 1 -- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 msgr2=0x7fc7481e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.638+0000 7fc74e400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74e400640 1 -- 172.21.15.92:0/1661882594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 msgr2=0x7fc748071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74e400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 0x7fc748071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74e400640 1 -- 172.21.15.92:0/1661882594 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7481f04d0 con 0x7fc7481593d0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74da00640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74d000640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 0x7fc748071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc74e400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7400097d0 tx=0x7fc7400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740037040 con 0x7fc7481593d0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc740032a40 con 0x7fc7481593d0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74814cf30 con 0x7fc7481593d0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740032e30 con 0x7fc7481593d0 2024-03-19T03:04:55.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.639+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74814d4f0 con 0x7fc7481593d0 2024-03-19T03:04:55.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.640+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7480684c0 con 0x7fc7481593d0 2024-03-19T03:04:55.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.641+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc740004030 con 0x7fc7481593d0 2024-03-19T03:04:55.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.642+0000 7fc745600640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 0x7fc72c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:55.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.642+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fc7400ac8a0 con 0x7fc7481593d0 2024-03-19T03:04:55.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.642+0000 7fc74da00640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 0x7fc72c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:55.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.643+0000 7fc74da00640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 0x7fc72c06e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007f70 tx=0x7fc73c007f00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:55.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.647+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc740042020 con 0x7fc7481593d0 2024-03-19T03:04:55.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.926+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc74806f950 con 0x7fc7481593d0 2024-03-19T03:04:55.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.927+0000 7fc745600640 1 -- 172.21.15.92:0/1661882594 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fc74007ab40 con 0x7fc7481593d0 2024-03-19T03:04:55.933 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:04:55.933 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:04:55.934 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:04:55.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.936+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 msgr2=0x7fc72c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.937+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 0x7fc72c06e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007f70 tx=0x7fc73c007f00 comp rx=0 tx=0).stop 2024-03-19T03:04:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.937+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 msgr2=0x7fc74814e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.937+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7400097d0 tx=0x7fc7400046c0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 shutdown_connections 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc72c06bbd0 0x7fc72c06e090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7481593d0 0x7fc74814e290 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc748152bf0 0x7fc748071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 --2- 172.21.15.92:0/1661882594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7480c6470 0x7fc7481e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 >> 172.21.15.92:0/1661882594 conn(0x7fc7480c8d90 msgr2=0x7fc74806eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 shutdown_connections 2024-03-19T03:04:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:55.938+0000 7fc750400640 1 -- 172.21.15.92:0/1661882594 wait complete. 2024-03-19T03:04:55.999 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2024-03-19T03:04:55.999 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-19T03:04:55.999 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:04:56.019 INFO:tasks.ceph.mds.d:Stopped 2024-03-19T03:04:56.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:56.217 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.216+0000 7ff61ee00640 1 Processor -- start 2024-03-19T03:04:56.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.216+0000 7ff61ee00640 1 -- start start 2024-03-19T03:04:56.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ee00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 0x7ff6180c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ee00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6180c9a80 con 0x7ff6180c6470 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ee00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff6180c6c90 con 0x7ff618152bf0 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff618159910 con 0x7ff6181593d0 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff617e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff61ce00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 0x7ff6180c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff60f400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff6180c6470 0x7ff6180c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:48948/0 (socket says 172.21.15.92:48948) 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff60f400640 1 -- 172.21.15.92:0/1398545732 learned_addr learned my addr 172.21.15.92:0/1398545732 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.217+0000 7ff617e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40432/0 (socket says 172.21.15.92:40432) 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff61ce00640 1 -- 172.21.15.92:0/1398545732 >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 msgr2=0x7ff61806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff61ce00640 1 --2- 172.21.15.92:0/1398545732 >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff61ce00640 1 -- 172.21.15.92:0/1398545732 >> v1:172.21.15.159:6789/0 conn(0x7ff6180c6470 legacy=0x7ff6180c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff61ce00640 1 -- 172.21.15.92:0/1398545732 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6180c8650 con 0x7ff6181593d0 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff617400640 1 -- 172.21.15.92:0/1398545732 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974146048 0 0) 0x7ff6180c9a80 con 0x7ff6180c6470 2024-03-19T03:04:56.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.218+0000 7ff617e00640 1 --2- 172.21.15.92:0/1398545732 >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.219+0000 7ff61ce00640 1 --2- 172.21.15.92:0/1398545732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 0x7ff6180c4690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff61002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789fcfc4a585a1ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.219+0000 7ff617400640 1 -- 172.21.15.92:0/1398545732 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610039070 con 0x7ff6181593d0 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.219+0000 7ff617400640 1 -- 172.21.15.92:0/1398545732 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff61002cbd0 con 0x7ff6181593d0 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.219+0000 7ff617400640 1 -- 172.21.15.92:0/1398545732 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610034440 con 0x7ff6181593d0 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 -- 172.21.15.92:0/1398545732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 msgr2=0x7ff6180c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 --2- 172.21.15.92:0/1398545732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 0x7ff6180c4690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff61002be20 comp rx=0 tx=0).stop 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 -- 172.21.15.92:0/1398545732 shutdown_connections 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 --2- 172.21.15.92:0/1398545732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6181593d0 0x7ff6180c4690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 --2- 172.21.15.92:0/1398545732 >> v2:172.21.15.92:3300/0 conn(0x7ff618152bf0 0x7ff61806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 -- 172.21.15.92:0/1398545732 >> 172.21.15.92:0/1398545732 conn(0x7ff6180c8d90 msgr2=0x7ff6180c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:56.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 -- 172.21.15.92:0/1398545732 shutdown_connections 2024-03-19T03:04:56.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.220+0000 7ff61ee00640 1 -- 172.21.15.92:0/1398545732 wait complete. 2024-03-19T03:04:56.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.222+0000 7ff61ee00640 1 Processor -- start 2024-03-19T03:04:56.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.222+0000 7ff61ee00640 1 -- start start 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff618152bf0 0x7ff6181f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6181593d0 0x7ff61815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff61806ea60 con 0x7ff6180c6470 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff61806e4a0 con 0x7ff6181593d0 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ee00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff61806e6f0 con 0x7ff618152bf0 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40436/0 (socket says 172.21.15.92:40436) 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 -- 172.21.15.92:0/2358514209 learned_addr learned my addr 172.21.15.92:0/2358514209 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff60f400640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff618152bf0 0x7ff6181f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff61ce00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6181593d0 0x7ff61815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 -- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff618152bf0 msgr2=0x7ff6181f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff618152bf0 0x7ff6181f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 -- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6181593d0 msgr2=0x7ff61815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6181593d0 0x7ff61815add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.223+0000 7ff617e00640 1 -- 172.21.15.92:0/2358514209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6181f0a90 con 0x7ff6180c6470 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff617e00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff60800ec10 tx=0x7ff60800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608017070 con 0x7ff6180c6470 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff60800cc30 con 0x7ff6180c6470 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608013470 con 0x7ff6180c6470 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6181f0d70 con 0x7ff6180c6470 2024-03-19T03:04:56.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.224+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61815a2f0 con 0x7ff6180c6470 2024-03-19T03:04:56.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.226+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6180684c0 con 0x7ff6180c6470 2024-03-19T03:04:56.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.226+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff60801a070 con 0x7ff6180c6470 2024-03-19T03:04:56.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.227+0000 7ff614c00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:56.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.227+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7ff60808ac30 con 0x7ff6180c6470 2024-03-19T03:04:56.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.227+0000 7ff60f400640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:56.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.228+0000 7ff60f400640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 0x7ff5f406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff6180c8580 tx=0x7ff600007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:56.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.232+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff608058db0 con 0x7ff6180c6470 2024-03-19T03:04:56.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.463+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff61806f950 con 0x7ff6180c6470 2024-03-19T03:04:56.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.464+0000 7ff614c00640 1 -- 172.21.15.92:0/2358514209 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7ff60805ce50 con 0x7ff6180c6470 2024-03-19T03:04:56.466 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:04:56.466 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:04:56.466 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:04:56.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.466+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 msgr2=0x7ff5f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.466+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 0x7ff5f406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff6180c8580 tx=0x7ff600007d50 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.466+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 msgr2=0x7ff6181f4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff60800ec10 tx=0x7ff60800c520 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 shutdown_connections 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6181593d0 0x7ff61815add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff618152bf0 0x7ff6181f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 --2- 172.21.15.92:0/2358514209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6180c6470 0x7ff6181f4090 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 >> 172.21.15.92:0/2358514209 conn(0x7ff6180c8d90 msgr2=0x7ff6180c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 shutdown_connections 2024-03-19T03:04:56.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:56.467+0000 7ff61ee00640 1 -- 172.21.15.92:0/2358514209 wait complete. 2024-03-19T03:04:56.486 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:04:58.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:58.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.701+0000 7fdc28a00640 1 Processor -- start 2024-03-19T03:04:58.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.701+0000 7fdc28a00640 1 -- start start 2024-03-19T03:04:58.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.702+0000 7fdc28a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdc240c6470 0x7fdc240c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.702+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 0x7fdc240c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.702+0000 7fdc28a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc240c9a80 con 0x7fdc24152bf0 2024-03-19T03:04:58.704 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.702+0000 7fdc28a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fdc240c6c90 con 0x7fdc240c6470 2024-03-19T03:04:58.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.702+0000 7fdc28a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdc24159910 con 0x7fdc241593d0 2024-03-19T03:04:58.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdc24152bf0 0x7fdc2406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58300/0 (socket says 172.21.15.92:58300) 2024-03-19T03:04:58.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21e00640 1 -- 172.21.15.92:0/3905681845 learned_addr learned my addr 172.21.15.92:0/3905681845 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:04:58.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc22800640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 0x7fdc240c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21400640 1 --2- 172.21.15.92:0/3905681845 >> v2:172.21.15.159:3300/0 conn(0x7fdc240c6470 0x7fdc240c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc20a00640 1 -- 172.21.15.92:0/3905681845 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651740575 0 0) 0x7fdc240c9a80 con 0x7fdc24152bf0 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc20a00640 1 -- 172.21.15.92:0/3905681845 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc08003520 con 0x7fdc24152bf0 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21400640 1 -- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 msgr2=0x7fdc240c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21400640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 0x7fdc240c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21400640 1 -- 172.21.15.92:0/3905681845 >> v1:172.21.15.92:6789/0 conn(0x7fdc24152bf0 legacy=0x7fdc2406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.703+0000 7fdc21400640 1 -- 172.21.15.92:0/3905681845 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc240c8650 con 0x7fdc240c6470 2024-03-19T03:04:58.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.704+0000 7fdc22800640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 0x7fdc240c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:58.707 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.705+0000 7fdc21400640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc240c6470 0x7fdc240c3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdc100097a0 tx=0x7fdc1002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3ff1ab3b26aebeb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:58.707 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.705+0000 7fdc20a00640 1 -- 172.21.15.92:0/3905681845 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10031780 con 0x7fdc240c6470 2024-03-19T03:04:58.707 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.705+0000 7fdc20a00640 1 -- 172.21.15.92:0/3905681845 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc10037070 con 0x7fdc240c6470 2024-03-19T03:04:58.707 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.705+0000 7fdc20a00640 1 -- 172.21.15.92:0/3905681845 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10031b20 con 0x7fdc240c6470 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 -- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc240c6470 msgr2=0x7fdc240c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc240c6470 0x7fdc240c3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdc100097a0 tx=0x7fdc1002b820 comp rx=0 tx=0).stop 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 -- 172.21.15.92:0/3905681845 shutdown_connections 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc241593d0 0x7fdc240c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3905681845 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc240c6470 0x7fdc240c3f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 -- 172.21.15.92:0/3905681845 >> 172.21.15.92:0/3905681845 conn(0x7fdc240c8d90 msgr2=0x7fdc240c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 -- 172.21.15.92:0/3905681845 shutdown_connections 2024-03-19T03:04:58.708 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.706+0000 7fdc28a00640 1 -- 172.21.15.92:0/3905681845 wait complete. 2024-03-19T03:04:58.709 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.708+0000 7fdc28a00640 1 Processor -- start 2024-03-19T03:04:58.709 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.708+0000 7fdc28a00640 1 -- start start 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 0x7fdc240cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 0x7fdc24146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406ea60 con 0x7fdc240c6470 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406e4a0 con 0x7fdc241593d0 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc28a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2406e6f0 con 0x7fdc24152bf0 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40446/0 (socket says 172.21.15.92:40446) 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 -- 172.21.15.92:0/3850076704 learned_addr learned my addr 172.21.15.92:0/3850076704 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21400640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 0x7fdc240cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc22800640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 0x7fdc24146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 -- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 msgr2=0x7fdc240cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.712 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 0x7fdc240cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 -- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 msgr2=0x7fdc24146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 0x7fdc24146fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21e00640 1 -- 172.21.15.92:0/3850076704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2415c680 con 0x7fdc240c6470 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc21400640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 0x7fdc240cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.709+0000 7fdc22800640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 0x7fdc24146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc21e00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c460 tx=0x7fdc1400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14017070 con 0x7fdc240c6470 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc14005ce0 con 0x7fdc240c6470 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14007290 con 0x7fdc240c6470 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2414fd20 con 0x7fdc240c6470 2024-03-19T03:04:58.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.710+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2415c3a0 con 0x7fdc240c6470 2024-03-19T03:04:58.714 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.711+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc24068de0 con 0x7fdc240c6470 2024-03-19T03:04:58.714 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.712+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdc1401a070 con 0x7fdc240c6470 2024-03-19T03:04:58.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.713+0000 7fdc1a000640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 0x7fdc0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:58.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.713+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fdc14089ff0 con 0x7fdc240c6470 2024-03-19T03:04:58.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.713+0000 7fdc21400640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 0x7fdc0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:58.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.714+0000 7fdc21400640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 0x7fdc0406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036000 tx=0x7fdc10008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:58.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.718+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdc24068de0 con 0x7fdc240c6470 2024-03-19T03:04:58.957 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.956+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc240684c0 con 0x7fdc240c6470 2024-03-19T03:04:58.958 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.957+0000 7fdc1a000640 1 -- 172.21.15.92:0/3850076704 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fdc140588f0 con 0x7fdc240c6470 2024-03-19T03:04:58.958 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:04:58.959 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:04:58.959 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 msgr2=0x7fdc0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 0x7fdc0406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036000 tx=0x7fdc10008030 comp rx=0 tx=0).stop 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 msgr2=0x7fdc240cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c460 tx=0x7fdc1400c920 comp rx=0 tx=0).stop 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 shutdown_connections 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdc0406bbd0 0x7fdc0406e090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc241593d0 0x7fdc24146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc24152bf0 0x7fdc240cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.961 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 --2- 172.21.15.92:0/3850076704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc240c6470 0x7fdc240cb190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:58.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 >> 172.21.15.92:0/3850076704 conn(0x7fdc240c8d90 msgr2=0x7fdc240c47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:58.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 shutdown_connections 2024-03-19T03:04:58.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:04:58.959+0000 7fdc28a00640 1 -- 172.21.15.92:0/3850076704 wait complete. 2024-03-19T03:04:58.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:05:00.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:01.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.189+0000 7fd2b0800640 1 Processor -- start 2024-03-19T03:05:01.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.189+0000 7fd2b0800640 1 -- start start 2024-03-19T03:05:01.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.190+0000 7fd2b0800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd2a80c60d0 0x7fd2a8071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:01.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.190+0000 7fd2b0800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a815b740 con 0x7fd2a8072380 2024-03-19T03:05:01.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2b0800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a815ba20 con 0x7fd2a8154dc0 2024-03-19T03:05:01.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2b0800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd2a80c8400 con 0x7fd2a80c60d0 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ad400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd2a8072380 0x7fd2a80c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58312/0 (socket says 172.21.15.92:58312) 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ad400640 1 -- 172.21.15.92:0/2005324780 learned_addr learned my addr 172.21.15.92:0/2005324780 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ae800640 1 --2- 172.21.15.92:0/2005324780 >> v2:172.21.15.159:3300/0 conn(0x7fd2a80c60d0 0x7fd2a8071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ae800640 1 -- 172.21.15.92:0/2005324780 >> v1:172.21.15.159:6789/0 conn(0x7fd2a8154dc0 legacy=0x7fd2a8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ae800640 1 -- 172.21.15.92:0/2005324780 >> v1:172.21.15.92:6789/0 conn(0x7fd2a8072380 legacy=0x7fd2a80c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2ae800640 1 -- 172.21.15.92:0/2005324780 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a80c86e0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.191+0000 7fd2aca00640 1 -- 172.21.15.92:0/2005324780 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416500815 0 0) 0x7fd2a815b740 con 0x7fd2a8072380 2024-03-19T03:05:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.193+0000 7fd2ae800640 1 --2- 172.21.15.92:0/2005324780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a80c60d0 0x7fd2a8071190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21fcd44933b75e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.193+0000 7fd2aca00640 1 -- 172.21.15.92:0/2005324780 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4039070 con 0x7fd2a80c60d0 2024-03-19T03:05:01.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.193+0000 7fd2aca00640 1 -- 172.21.15.92:0/2005324780 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a402cc60 con 0x7fd2a80c60d0 2024-03-19T03:05:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.193+0000 7fd2aca00640 1 -- 172.21.15.92:0/2005324780 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4034660 con 0x7fd2a80c60d0 2024-03-19T03:05:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 -- 172.21.15.92:0/2005324780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a80c60d0 msgr2=0x7fd2a8071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 --2- 172.21.15.92:0/2005324780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a80c60d0 0x7fd2a8071190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a402beb0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 -- 172.21.15.92:0/2005324780 shutdown_connections 2024-03-19T03:05:01.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 --2- 172.21.15.92:0/2005324780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a80c60d0 0x7fd2a8071190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 -- 172.21.15.92:0/2005324780 >> 172.21.15.92:0/2005324780 conn(0x7fd2a80c8d90 msgr2=0x7fd2a80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:01.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 -- 172.21.15.92:0/2005324780 shutdown_connections 2024-03-19T03:05:01.196 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.194+0000 7fd2b0800640 1 -- 172.21.15.92:0/2005324780 wait complete. 2024-03-19T03:05:01.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.195+0000 7fd2b0800640 1 Processor -- start 2024-03-19T03:05:01.197 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 -- start start 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2a8072380 0x7fd2a815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 0x7fd2a815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a80c7520 con 0x7fd2a80c60d0 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.196+0000 7fd2b0800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a80c73a0 con 0x7fd2a8154dc0 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2b0800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd2a80c76a0 con 0x7fd2a8072380 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40458/0 (socket says 172.21.15.92:40458) 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 -- 172.21.15.92:0/4265704578 learned_addr learned my addr 172.21.15.92:0/4265704578 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ae800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 0x7fd2a815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ade00640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2a8072380 0x7fd2a815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 -- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2a8072380 msgr2=0x7fd2a815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.200 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2a8072380 0x7fd2a815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 -- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 msgr2=0x7fd2a815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 0x7fd2a815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ad400640 1 -- 172.21.15.92:0/4265704578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a815be00 con 0x7fd2a80c60d0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.197+0000 7fd2ae800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 0x7fd2a815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2ad400640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294018070 con 0x7fd2a80c60d0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2940074e0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294004a20 con 0x7fd2a80c60d0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a81f0030 con 0x7fd2a80c60d0 2024-03-19T03:05:01.201 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.198+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a81f05d0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.199+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a80684c0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.200+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd294004bc0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.202 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.201+0000 7fd2a2000640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 0x7fd28806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.201+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fd294089e90 con 0x7fd2a80c60d0 2024-03-19T03:05:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.201+0000 7fd2ade00640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 0x7fd28806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.202+0000 7fd2ade00640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 0x7fd28806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd29c000f60 tx=0x7fd29c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.207+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd294058010 con 0x7fd2a80c60d0 2024-03-19T03:05:01.446 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.444+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2a80c7cb0 con 0x7fd2a80c60d0 2024-03-19T03:05:01.447 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.446+0000 7fd2a2000640 1 -- 172.21.15.92:0/4265704578 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fd29408b090 con 0x7fd2a80c60d0 2024-03-19T03:05:01.448 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:01.448 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:05:01.448 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:05:01.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 msgr2=0x7fd28806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 0x7fd28806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd29c000f60 tx=0x7fd29c009040 comp rx=0 tx=0).stop 2024-03-19T03:05:01.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 msgr2=0x7fd2a815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c990 comp rx=0 tx=0).stop 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 shutdown_connections 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd28806bbd0 0x7fd28806e090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2a8154dc0 0x7fd2a815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a80c60d0 0x7fd2a815c780 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 --2- 172.21.15.92:0/4265704578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd2a8072380 0x7fd2a815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.448+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 >> 172.21.15.92:0/4265704578 conn(0x7fd2a80c8d90 msgr2=0x7fd2a80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.449+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 shutdown_connections 2024-03-19T03:05:01.451 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:01.449+0000 7fd2b0800640 1 -- 172.21.15.92:0/4265704578 wait complete. 2024-03-19T03:05:01.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:05:01.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:05:01.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:05:03.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:03.685 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.684+0000 7fbc9d800640 1 Processor -- start 2024-03-19T03:05:03.685 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.684+0000 7fbc9d800640 1 -- start start 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc9d800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc980c6470 0x7fbc980c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc9d800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbc981593d0 0x7fbc980c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc9d800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc980c9a80 con 0x7fbc98152bf0 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc9d800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbc980c6c90 con 0x7fbc980c6470 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc9d800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fbc98159910 con 0x7fbc981593d0 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbc98152bf0 0x7fbc9806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:58320/0 (socket says 172.21.15.92:58320) 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96a00640 1 -- 172.21.15.92:0/977194086 learned_addr learned my addr 172.21.15.92:0/977194086 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:03.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96000640 1 --2- 172.21.15.92:0/977194086 >> v2:172.21.15.92:3300/0 conn(0x7fbc980c6470 0x7fbc980c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc97400640 1 --2- 172.21.15.92:0/977194086 >> v2:172.21.15.159:3300/0 conn(0x7fbc981593d0 0x7fbc980c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc95600640 1 -- 172.21.15.92:0/977194086 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649140353 0 0) 0x7fbc980c9a80 con 0x7fbc98152bf0 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc95600640 1 -- 172.21.15.92:0/977194086 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc7c003520 con 0x7fbc98152bf0 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96000640 1 -- 172.21.15.92:0/977194086 >> v2:172.21.15.159:3300/0 conn(0x7fbc981593d0 msgr2=0x7fbc980c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96000640 1 --2- 172.21.15.92:0/977194086 >> v2:172.21.15.159:3300/0 conn(0x7fbc981593d0 0x7fbc980c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96000640 1 -- 172.21.15.92:0/977194086 >> v1:172.21.15.92:6789/0 conn(0x7fbc98152bf0 legacy=0x7fbc9806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.686+0000 7fbc96000640 1 -- 172.21.15.92:0/977194086 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc980c8650 con 0x7fbc980c6470 2024-03-19T03:05:03.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc96000640 1 --2- 172.21.15.92:0/977194086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc980c6470 0x7fbc980c3f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbc980c5070 tx=0x7fbc8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6b68a5779a56c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:03.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc95600640 1 -- 172.21.15.92:0/977194086 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc840315a0 con 0x7fbc980c6470 2024-03-19T03:05:03.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc95600640 1 -- 172.21.15.92:0/977194086 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc84037070 con 0x7fbc980c6470 2024-03-19T03:05:03.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc95600640 1 -- 172.21.15.92:0/977194086 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84031940 con 0x7fbc980c6470 2024-03-19T03:05:03.689 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 -- 172.21.15.92:0/977194086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc980c6470 msgr2=0x7fbc980c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 --2- 172.21.15.92:0/977194086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc980c6470 0x7fbc980c3f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbc980c5070 tx=0x7fbc8402b840 comp rx=0 tx=0).stop 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 -- 172.21.15.92:0/977194086 shutdown_connections 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 --2- 172.21.15.92:0/977194086 >> v2:172.21.15.159:3300/0 conn(0x7fbc981593d0 0x7fbc980c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 --2- 172.21.15.92:0/977194086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc980c6470 0x7fbc980c3f80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 -- 172.21.15.92:0/977194086 >> 172.21.15.92:0/977194086 conn(0x7fbc980c8d90 msgr2=0x7fbc980c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.688+0000 7fbc9d800640 1 -- 172.21.15.92:0/977194086 shutdown_connections 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.689+0000 7fbc9d800640 1 -- 172.21.15.92:0/977194086 wait complete. 2024-03-19T03:05:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.690+0000 7fbc9d800640 1 Processor -- start 2024-03-19T03:05:03.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.690+0000 7fbc9d800640 1 -- start start 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 0x7fbc98071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 0x7fbc980716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9806ea60 con 0x7fbc981593d0 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9806e4a0 con 0x7fbc98152bf0 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc9d800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9806e6f0 con 0x7fbc980c6470 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc96a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc96000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 0x7fbc98071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.692 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc96a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42276/0 (socket says 172.21.15.92:42276) 2024-03-19T03:05:03.693 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.691+0000 7fbc96a00640 1 -- 172.21.15.92:0/4053273312 learned_addr learned my addr 172.21.15.92:0/4053273312 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:03.693 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc97400640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 0x7fbc980716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 -- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 msgr2=0x7fbc98071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 0x7fbc98071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 -- 172.21.15.92:0/4053273312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 msgr2=0x7fbc980716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 0x7fbc980716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 -- 172.21.15.92:0/4053273312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc981f02d0 con 0x7fbc980c6470 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96000640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 0x7fbc98071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc97400640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 0x7fbc980716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc96a00640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800c460 tx=0x7fbc8800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.692+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88017070 con 0x7fbc980c6470 2024-03-19T03:05:03.694 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.693+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc88005ce0 con 0x7fbc980c6470 2024-03-19T03:05:03.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.693+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc980cba20 con 0x7fbc980c6470 2024-03-19T03:05:03.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.693+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88007290 con 0x7fbc980c6470 2024-03-19T03:05:03.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.693+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc981f0820 con 0x7fbc980c6470 2024-03-19T03:05:03.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.694+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc981f0e30 con 0x7fbc980c6470 2024-03-19T03:05:03.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.695+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbc8801a070 con 0x7fbc980c6470 2024-03-19T03:05:03.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.695+0000 7fbc8ea00640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 0x7fbc7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:03.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.696+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fbc88089fb0 con 0x7fbc980c6470 2024-03-19T03:05:03.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.696+0000 7fbc96000640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 0x7fbc7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:03.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.696+0000 7fbc96000640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 0x7fbc7806e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbc84036000 tx=0x7fbc8403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:03.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.701+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbc880581d0 con 0x7fbc980c6470 2024-03-19T03:05:03.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.917+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc980684c0 con 0x7fbc980c6470 2024-03-19T03:05:03.918 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.918+0000 7fbc8ea00640 1 -- 172.21.15.92:0/4053273312 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fbc8805c270 con 0x7fbc980c6470 2024-03-19T03:05:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:03.919 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:05:03.919 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:05:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 msgr2=0x7fbc7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 0x7fbc7806e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbc84036000 tx=0x7fbc8403d040 comp rx=0 tx=0).stop 2024-03-19T03:05:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 msgr2=0x7fbc981e5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800c460 tx=0x7fbc8800c920 comp rx=0 tx=0).stop 2024-03-19T03:05:03.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 shutdown_connections 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbc7806bbd0 0x7fbc7806e090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc981593d0 0x7fbc980716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc98152bf0 0x7fbc98071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 --2- 172.21.15.92:0/4053273312 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc980c6470 0x7fbc981e5230 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.920+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 >> 172.21.15.92:0/4053273312 conn(0x7fbc980c8d90 msgr2=0x7fbc98150ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.921+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 shutdown_connections 2024-03-19T03:05:03.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:03.921+0000 7fbc9d800640 1 -- 172.21.15.92:0/4053273312 wait complete. 2024-03-19T03:05:03.939 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:05:05.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:06.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.156+0000 7f0d73200640 1 Processor -- start 2024-03-19T03:05:06.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.156+0000 7f0d73200640 1 -- start start 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.157+0000 7f0d73200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 0x7f0d74151610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.157+0000 7f0d73200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d74072140 con 0x7f0d740c5480 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.157+0000 7f0d73200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d740c4ef0 con 0x7f0d740c5860 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.157+0000 7f0d73200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d74159520 con 0x7f0d7415a0c0 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d71e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0d740c5480 0x7f0d74072390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40478/0 (socket says 172.21.15.92:40478) 2024-03-19T03:05:06.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d71e00640 1 -- 172.21.15.92:0/843445396 learned_addr learned my addr 172.21.15.92:0/843445396 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:06.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d72800640 1 --2- 172.21.15.92:0/843445396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 0x7f0d74151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:06.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404379199 0 0) 0x7f0d74072140 con 0x7f0d740c5480 2024-03-19T03:05:06.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d58005730 con 0x7f0d740c5480 2024-03-19T03:05:06.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574757630 0 0) 0x7f0d5c001110 con 0x7f0d740c5860 2024-03-19T03:05:06.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d72800640 1 -- 172.21.15.92:0/843445396 >> v1:172.21.15.159:6789/0 conn(0x7f0d740c5860 legacy=0x7f0d741599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d72800640 1 -- 172.21.15.92:0/843445396 >> v1:172.21.15.92:6789/0 conn(0x7f0d740c5480 legacy=0x7f0d74072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.158+0000 7f0d72800640 1 -- 172.21.15.92:0/843445396 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d74071b80 con 0x7f0d7415a0c0 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.159+0000 7f0d72800640 1 --2- 172.21.15.92:0/843445396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 0x7f0d74151610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0097d0 tx=0x7f0d6c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f708bd66617602a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c035020 con 0x7f0d7415a0c0 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d6c031800 con 0x7f0d7415a0c0 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d71400640 1 -- 172.21.15.92:0/843445396 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c031bd0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 -- 172.21.15.92:0/843445396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 msgr2=0x7f0d74151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 --2- 172.21.15.92:0/843445396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 0x7f0d74151610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0097d0 tx=0x7f0d6c02b8a0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 -- 172.21.15.92:0/843445396 shutdown_connections 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 --2- 172.21.15.92:0/843445396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d7415a0c0 0x7f0d74151610 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 -- 172.21.15.92:0/843445396 >> 172.21.15.92:0/843445396 conn(0x7f0d740cc1f0 msgr2=0x7f0d740cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 -- 172.21.15.92:0/843445396 shutdown_connections 2024-03-19T03:05:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.160+0000 7f0d73200640 1 -- 172.21.15.92:0/843445396 wait complete. 2024-03-19T03:05:06.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.162+0000 7f0d73200640 1 Processor -- start 2024-03-19T03:05:06.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.162+0000 7f0d73200640 1 -- start start 2024-03-19T03:05:06.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 0x7f0d741f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d740c5860 0x7f0d741e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 0x7f0d74145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0d741591b0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0d74158ed0 con 0x7f0d740c5860 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d73200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d740c66a0 con 0x7f0d740c5480 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 0x7f0d741f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d72800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 0x7f0d74145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d69400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d740c5860 0x7f0d741e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 0x7f0d741f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:47172/0 (socket says 172.21.15.92:47172) 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.163+0000 7f0d71e00640 1 -- 172.21.15.92:0/2592572637 learned_addr learned my addr 172.21.15.92:0/2592572637 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 -- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 msgr2=0x7f0d741f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 0x7f0d741f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 -- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d740c5860 msgr2=0x7f0d741e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d740c5860 0x7f0d741e5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 -- 172.21.15.92:0/2592572637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d740c9b70 con 0x7f0d7415a0c0 2024-03-19T03:05:06.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d72800640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 0x7f0d74145aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c033000 tx=0x7f0d6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:06.166 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.164+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c040070 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.165+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d6c039bf0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.165+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c035040 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.165+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d741e3270 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.165+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d741e3830 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.166+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d741e41d0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.167+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0d6c038020 con 0x7f0d7415a0c0 2024-03-19T03:05:06.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.168+0000 7f0d6aa00640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 0x7f0d5006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:06.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.168+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7f0d6c0ac5e0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.168+0000 7f0d71e00640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 0x7f0d5006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:06.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.169+0000 7f0d71e00640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 0x7f0d5006e080 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d64005c00 tx=0x7f0d64009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:06.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.172+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0d741e41d0 con 0x7f0d7415a0c0 2024-03-19T03:05:06.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.403+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d7404f580 con 0x7f0d7415a0c0 2024-03-19T03:05:06.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.404+0000 7f0d6aa00640 1 -- 172.21.15.92:0/2592572637 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f0d6c07a890 con 0x7f0d7415a0c0 2024-03-19T03:05:06.405 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:06.405 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:05:06.406 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:05:06.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 msgr2=0x7f0d5006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 0x7f0d5006e080 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d64005c00 tx=0x7f0d64009040 comp rx=0 tx=0).stop 2024-03-19T03:05:06.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 msgr2=0x7f0d74145aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:06.408 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 0x7f0d74145aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c033000 tx=0x7f0d6c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 shutdown_connections 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f0d5006bbc0 0x7f0d5006e080 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d7415a0c0 0x7f0d74145aa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.407+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d740c5860 0x7f0d741e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.408+0000 7f0d73200640 1 --2- 172.21.15.92:0/2592572637 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d740c5480 0x7f0d741f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.408+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 >> 172.21.15.92:0/2592572637 conn(0x7f0d740cc1f0 msgr2=0x7f0d7406dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.408+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 shutdown_connections 2024-03-19T03:05:06.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:06.408+0000 7f0d73200640 1 -- 172.21.15.92:0/2592572637 wait complete. 2024-03-19T03:05:06.427 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:05:08.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:08.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.647+0000 7fc570000640 1 Processor -- start 2024-03-19T03:05:08.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.647+0000 7fc570000640 1 -- start start 2024-03-19T03:05:08.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.648+0000 7fc570000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 0x7fc5680c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.648+0000 7fc570000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc5681593d0 0x7fc5680c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc570000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5680c9a80 con 0x7fc568152bf0 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc570000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fc5680c6c90 con 0x7fc5681593d0 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc570000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc568159910 con 0x7fc5680c6470 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc5681593d0 0x7fc5680c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56cc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 0x7fc5680c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc5681593d0 0x7fc5680c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:49734/0 (socket says 172.21.15.92:49734) 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 -- 172.21.15.92:0/690384073 learned_addr learned my addr 172.21.15.92:0/690384073 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 -- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 msgr2=0x7fc5680c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:08.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 0x7fc5680c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 -- 172.21.15.92:0/690384073 >> v1:172.21.15.159:6789/0 conn(0x7fc568152bf0 legacy=0x7fc56806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56e000640 1 -- 172.21.15.92:0/690384073 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5680c8650 con 0x7fc5681593d0 2024-03-19T03:05:08.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.649+0000 7fc56cc00640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 0x7fc5680c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.650+0000 7fc56e000640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5681593d0 0x7fc5680c44c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc560009e50 tx=0x7fc56002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7dc694a4fd2717 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc567e00640 1 -- 172.21.15.92:0/690384073 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56002ca80 con 0x7fc5681593d0 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc567e00640 1 -- 172.21.15.92:0/690384073 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56002cc20 con 0x7fc5681593d0 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc567e00640 1 -- 172.21.15.92:0/690384073 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5600343f0 con 0x7fc5681593d0 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 -- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5681593d0 msgr2=0x7fc5680c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5681593d0 0x7fc5680c44c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc560009e50 tx=0x7fc56002bee0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 -- 172.21.15.92:0/690384073 shutdown_connections 2024-03-19T03:05:08.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5680c6470 0x7fc5680c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 --2- 172.21.15.92:0/690384073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5681593d0 0x7fc5680c44c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.651+0000 7fc570000640 1 -- 172.21.15.92:0/690384073 >> 172.21.15.92:0/690384073 conn(0x7fc5680c8d90 msgr2=0x7fc5680c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:08.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.652+0000 7fc570000640 1 -- 172.21.15.92:0/690384073 shutdown_connections 2024-03-19T03:05:08.654 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.652+0000 7fc570000640 1 -- 172.21.15.92:0/690384073 wait complete. 2024-03-19T03:05:08.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.654+0000 7fc570000640 1 Processor -- start 2024-03-19T03:05:08.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.654+0000 7fc570000640 1 -- start start 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5680c6470 0x7fc56814dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 0x7fc5681418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc56806ea60 con 0x7fc568152bf0 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc56806e4a0 con 0x7fc5680c6470 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc570000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc56806e6f0 con 0x7fc5681593d0 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 0x7fc5681418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56d600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5680c6470 0x7fc56814dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.658 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56e000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:47180/0 (socket says 172.21.15.92:47180) 2024-03-19T03:05:08.658 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56e000640 1 -- 172.21.15.92:0/2923651078 learned_addr learned my addr 172.21.15.92:0/2923651078 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:08.658 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.655+0000 7fc56cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 0x7fc5681418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36266/0 (socket says 172.21.15.92:36266) 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 -- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5680c6470 msgr2=0x7fc56814dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5680c6470 0x7fc56814dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 -- 172.21.15.92:0/2923651078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 msgr2=0x7fc5681418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 0x7fc5681418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 -- 172.21.15.92:0/2923651078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc568141e20 con 0x7fc5681593d0 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc56e000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc560036040 tx=0x7fc560004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560043020 con 0x7fc5681593d0 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56003e450 con 0x7fc5681593d0 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560009510 con 0x7fc5681593d0 2024-03-19T03:05:08.659 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.656+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56814e780 con 0x7fc5681593d0 2024-03-19T03:05:08.660 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.657+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5680ca4c0 con 0x7fc5681593d0 2024-03-19T03:05:08.660 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.658+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5680684c0 con 0x7fc5681593d0 2024-03-19T03:05:08.660 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.658+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc560046020 con 0x7fc5681593d0 2024-03-19T03:05:08.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.659+0000 7fc565600640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 0x7fc54806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:08.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.659+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6198+0+0 (secure 0 0 0) 0x7fc5600ad270 con 0x7fc5681593d0 2024-03-19T03:05:08.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.660+0000 7fc56d600640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 0x7fc54806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:08.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.660+0000 7fc56d600640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 0x7fc54806e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001780 tx=0x7fc55c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:08.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.664+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc56004d030 con 0x7fc5681593d0 2024-03-19T03:05:08.902 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.901+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5681f3ad0 con 0x7fc5681593d0 2024-03-19T03:05:08.903 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.902+0000 7fc565600640 1 -- 172.21.15.92:0/2923651078 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fc568141e20 con 0x7fc5681593d0 2024-03-19T03:05:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:08.904 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5347,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:03:53.149318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.159:6839/3845908330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3845908330},{"type":"v1","addr":"172.21.15.159:6839","nonce":3845908330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5185,"qdb_cluster":[5185]},"id":1}]} 2024-03-19T03:05:08.904 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 31 2024-03-19T03:05:08.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.904+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 msgr2=0x7fc54806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.904+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 0x7fc54806e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001780 tx=0x7fc55c009040 comp rx=0 tx=0).stop 2024-03-19T03:05:08.906 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.904+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 msgr2=0x7fc568147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.904+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc560036040 tx=0x7fc560004a40 comp rx=0 tx=0).stop 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 shutdown_connections 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fc54806bbd0 0x7fc54806e090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5681593d0 0x7fc568147190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc568152bf0 0x7fc5681418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 --2- 172.21.15.92:0/2923651078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5680c6470 0x7fc56814dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 >> 172.21.15.92:0/2923651078 conn(0x7fc5680c8d90 msgr2=0x7fc568148400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 shutdown_connections 2024-03-19T03:05:08.907 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:08.905+0000 7fc570000640 1 -- 172.21.15.92:0/2923651078 wait complete. 2024-03-19T03:05:08.925 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-19T03:05:10.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:11.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.145+0000 7f4ca7200640 1 Processor -- start 2024-03-19T03:05:11.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.145+0000 7f4ca7200640 1 -- start start 2024-03-19T03:05:11.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca7200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4ca81549c0 0x7f4ca80cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca7200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca815ac30 con 0x7f4ca80cc300 2024-03-19T03:05:11.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca7200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca815e280 con 0x7f4ca80b3180 2024-03-19T03:05:11.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca7200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca815e030 con 0x7f4ca81549c0 2024-03-19T03:05:11.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca6800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4ca80cc300 0x7f4ca81653c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40492/0 (socket says 172.21.15.92:40492) 2024-03-19T03:05:11.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca6800640 1 -- 172.21.15.92:0/2128152042 learned_addr learned my addr 172.21.15.92:0/2128152042 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.147+0000 7f4ca5e00640 1 --2- 172.21.15.92:0/2128152042 >> v2:172.21.15.92:3300/0 conn(0x7f4ca81549c0 0x7f4ca80cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786827543 0 0) 0x7f4ca815ac30 con 0x7f4ca80cc300 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c8c003520 con 0x7f4ca80cc300 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857838155 0 0) 0x7f4ca815e280 con 0x7f4ca80b3180 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ca815ac30 con 0x7f4ca80b3180 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca5e00640 1 -- 172.21.15.92:0/2128152042 >> v1:172.21.15.159:6789/0 conn(0x7f4ca80b3180 legacy=0x7f4ca80cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca5e00640 1 -- 172.21.15.92:0/2128152042 >> v1:172.21.15.92:6789/0 conn(0x7f4ca80cc300 legacy=0x7f4ca81653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.148+0000 7f4ca5e00640 1 -- 172.21.15.92:0/2128152042 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca815e560 con 0x7f4ca81549c0 2024-03-19T03:05:11.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.149+0000 7f4ca5e00640 1 --2- 172.21.15.92:0/2128152042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c464e7b067bd62d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.149+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98035020 con 0x7f4ca81549c0 2024-03-19T03:05:11.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.149+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c98031830 con 0x7f4ca81549c0 2024-03-19T03:05:11.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.149+0000 7f4ca4a00640 1 -- 172.21.15.92:0/2128152042 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98031c00 con 0x7f4ca81549c0 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 -- 172.21.15.92:0/2128152042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 msgr2=0x7f4ca80cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 --2- 172.21.15.92:0/2128152042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802b8d0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 -- 172.21.15.92:0/2128152042 shutdown_connections 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 --2- 172.21.15.92:0/2128152042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 -- 172.21.15.92:0/2128152042 >> 172.21.15.92:0/2128152042 conn(0x7f4ca8072970 msgr2=0x7f4ca8066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 -- 172.21.15.92:0/2128152042 shutdown_connections 2024-03-19T03:05:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.150+0000 7f4ca7200640 1 -- 172.21.15.92:0/2128152042 wait complete. 2024-03-19T03:05:11.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.151+0000 7f4ca7200640 1 Processor -- start 2024-03-19T03:05:11.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 -- start start 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ca80cc300 0x7f4ca8146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca815ee00 con 0x7f4ca81549c0 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca815eb20 con 0x7f4ca80cc300 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.152+0000 7f4ca7200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca8154da0 con 0x7f4ca80b3180 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ca80cc300 0x7f4ca8146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:47184/0 (socket says 172.21.15.92:47184) 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 -- 172.21.15.92:0/233242319 learned_addr learned my addr 172.21.15.92:0/233242319 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 -- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ca80cc300 msgr2=0x7f4ca8146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ca80cc300 0x7f4ca8146fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 -- 172.21.15.92:0/233242319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 msgr2=0x7f4ca80cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca5e00640 1 -- 172.21.15.92:0/233242319 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca815bc00 con 0x7f4ca80b3180 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.153+0000 7f4ca6800640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:11.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4ca5e00640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034040 tx=0x7f4c980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98040070 con 0x7f4ca80b3180 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9803a910 con 0x7f4ca80b3180 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca81f3d00 con 0x7f4ca80b3180 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98035040 con 0x7f4ca80b3180 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.154+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca80ca480 con 0x7f4ca80b3180 2024-03-19T03:05:11.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.155+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca804f580 con 0x7f4ca80b3180 2024-03-19T03:05:11.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.156+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4c98038020 con 0x7f4ca80b3180 2024-03-19T03:05:11.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.157+0000 7f4c9e000640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 0x7f4c8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.157+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f4c980ac350 con 0x7f4ca80b3180 2024-03-19T03:05:11.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.158+0000 7f4ca5400640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 0x7f4c8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.158+0000 7f4ca5400640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 0x7f4c8806e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80b5800 tx=0x7f4c94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.162+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4c9807a5a0 con 0x7f4ca80b3180 2024-03-19T03:05:11.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.401+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ca80cb2d0 con 0x7f4ca80b3180 2024-03-19T03:05:11.409 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.407+0000 7f4c9e000640 1 -- 172.21.15.92:0/233242319 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6003 (secure 0 0 0) 0x7f4c9807e640 con 0x7f4ca80b3180 2024-03-19T03:05:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:11.410 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:09.726694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:11.410 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 32 2024-03-19T03:05:11.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 msgr2=0x7f4c8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.412 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 0x7f4c8806e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80b5800 tx=0x7f4c94009040 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 msgr2=0x7f4ca80c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034040 tx=0x7f4c980046c0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 shutdown_connections 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c8806bbd0 0x7f4c8806e090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca81549c0 0x7f4ca80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ca80cc300 0x7f4ca8146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 --2- 172.21.15.92:0/233242319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ca80b3180 0x7f4ca80c9db0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.410+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 >> 172.21.15.92:0/233242319 conn(0x7f4ca8072970 msgr2=0x7f4ca80ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.411+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 shutdown_connections 2024-03-19T03:05:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.411+0000 7f4ca7200640 1 -- 172.21.15.92:0/233242319 wait complete. 2024-03-19T03:05:11.431 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-03-19T03:05:11.431 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-19T03:05:11.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:11.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.636+0000 7f00d3800640 1 Processor -- start 2024-03-19T03:05:11.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.636+0000 7f00d3800640 1 -- start start 2024-03-19T03:05:11.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.637+0000 7f00d3800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00cc158320 0x7f00cc163150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.637+0000 7f00d3800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00cc0c3b80 con 0x7f00cc151900 2024-03-19T03:05:11.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.637+0000 7f00d3800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00cc072e70 con 0x7f00cc151ce0 2024-03-19T03:05:11.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.637+0000 7f00d3800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f00cc073150 con 0x7f00cc158320 2024-03-19T03:05:11.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d0e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f00cc151900 0x7f00cc0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40498/0 (socket says 172.21.15.92:40498) 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d0e00640 1 -- 172.21.15.92:0/858161820 learned_addr learned my addr 172.21.15.92:0/858161820 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d1800640 1 --2- 172.21.15.92:0/858161820 >> v2:172.21.15.92:3300/0 conn(0x7f00cc158320 0x7f00cc163150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710937778 0 0) 0x7f00cc0c3b80 con 0x7f00cc151900 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00b0003520 con 0x7f00cc151900 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353479578 0 0) 0x7f00cc072e70 con 0x7f00cc151ce0 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00cc0c3b80 con 0x7f00cc151ce0 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d1800640 1 -- 172.21.15.92:0/858161820 >> v1:172.21.15.159:6789/0 conn(0x7f00cc151ce0 legacy=0x7f00cc0c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d1800640 1 -- 172.21.15.92:0/858161820 >> v1:172.21.15.92:6789/0 conn(0x7f00cc151900 legacy=0x7f00cc0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.638+0000 7f00d1800640 1 -- 172.21.15.92:0/858161820 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc071bd0 con 0x7f00cc158320 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.640+0000 7f00d1800640 1 --2- 172.21.15.92:0/858161820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc163150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009d90 tx=0x7f00c402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c490121c8aed2bda server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.640+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4039070 con 0x7f00cc158320 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.640+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c402cd80 con 0x7f00cc158320 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.640+0000 7f00cbe00640 1 -- 172.21.15.92:0/858161820 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4034660 con 0x7f00cc158320 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 -- 172.21.15.92:0/858161820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 msgr2=0x7f00cc163150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 --2- 172.21.15.92:0/858161820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc163150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009d90 tx=0x7f00c402bfd0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 -- 172.21.15.92:0/858161820 shutdown_connections 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 --2- 172.21.15.92:0/858161820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc163150 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 -- 172.21.15.92:0/858161820 >> 172.21.15.92:0/858161820 conn(0x7f00cc0c8d90 msgr2=0x7f00cc0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 -- 172.21.15.92:0/858161820 shutdown_connections 2024-03-19T03:05:11.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.641+0000 7f00d3800640 1 -- 172.21.15.92:0/858161820 wait complete. 2024-03-19T03:05:11.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.643+0000 7f00d3800640 1 Processor -- start 2024-03-19T03:05:11.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.643+0000 7f00d3800640 1 -- start start 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00cc151900 0x7f00cc156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00cc151ce0 0x7f00cc156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc0c9af0 con 0x7f00cc158320 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc158770 con 0x7f00cc151ce0 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d3800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f00cc0c9c70 con 0x7f00cc151900 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36312/0 (socket says 172.21.15.92:36312) 2024-03-19T03:05:11.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d1800640 1 -- 172.21.15.92:0/4167241174 learned_addr learned my addr 172.21.15.92:0/4167241174 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:11.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00d0e00640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00cc151900 0x7f00cc156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.644+0000 7f00c3e00640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00cc151ce0 0x7f00cc156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 -- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00cc151900 msgr2=0x7f00cc156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00cc151900 0x7f00cc156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 -- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00cc151ce0 msgr2=0x7f00cc156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00cc151ce0 0x7f00cc156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 -- 172.21.15.92:0/4167241174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc1560b0 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d1800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009eb0 tx=0x7f00c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4039050 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c4045070 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4009450 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc14daf0 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.645+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc14c540 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.647+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc0684c0 con 0x7f00cc158320 2024-03-19T03:05:11.649 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.647+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f00c403e850 con 0x7f00cc158320 2024-03-19T03:05:11.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.649+0000 7f00c9600640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 0x7f00a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:11.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.649+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f00c40adb90 con 0x7f00cc158320 2024-03-19T03:05:11.650 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.649+0000 7f00d0e00640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 0x7f00a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:11.651 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.650+0000 7f00d0e00640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 0x7f00a806e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000f60 tx=0x7f00bc007d70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:11.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.653+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f00c407bde0 con 0x7f00cc158320 2024-03-19T03:05:11.878 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.876+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00cc14e010 con 0x7f00cc158320 2024-03-19T03:05:11.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.877+0000 7f00c9600640 1 -- 172.21.15.92:0/4167241174 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6003 (secure 0 0 0) 0x7f00c407fe80 con 0x7f00cc158320 2024-03-19T03:05:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:11.879 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:09.726694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:11.879 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 32 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 msgr2=0x7f00a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 0x7f00a806e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000f60 tx=0x7f00bc007d70 comp rx=0 tx=0).stop 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 msgr2=0x7f00cc146720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009eb0 tx=0x7f00c4004a40 comp rx=0 tx=0).stop 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 shutdown_connections 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f00a806bbd0 0x7f00a806e090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00cc158320 0x7f00cc146720 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00cc151ce0 0x7f00cc156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 --2- 172.21.15.92:0/4167241174 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00cc151900 0x7f00cc156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:11.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 >> 172.21.15.92:0/4167241174 conn(0x7f00cc0c8d90 msgr2=0x7f00cc06df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:11.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 shutdown_connections 2024-03-19T03:05:11.883 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:11.880+0000 7f00d3800640 1 -- 172.21.15.92:0/4167241174 wait complete. 2024-03-19T03:05:11.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5347": { "addr": "172.21.15.159:6837/3372688839", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 3372688839, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 3372688839, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5347, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:05:09.726694+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5347 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:05:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:14.117 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.115+0000 7f18cda00640 1 Processor -- start 2024-03-19T03:05:14.117 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.115+0000 7f18cda00640 1 -- start start 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.116+0000 7f18cda00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 0x7f18c815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.116+0000 7f18cda00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.116+0000 7f18cda00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c80c6d00 con 0x7f18c806d160 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.116+0000 7f18cda00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f18c806df20 con 0x7f18c80c5b70 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.116+0000 7f18cda00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18c806d610 con 0x7f18c80c5f50 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18c7e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 0x7f18c815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:47206/0 (socket says 172.21.15.92:47206) 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 -- 172.21.15.92:0/4149997583 learned_addr learned my addr 172.21.15.92:0/4149997583 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18cd000640 1 --1- 172.21.15.92:0/4149997583 >> v1:172.21.15.159:6789/0 conn(0x7f18c806d160 0x7f18c8152a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:37528/0 (socket says 172.21.15.92:37528) 2024-03-19T03:05:14.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18c7400640 1 -- 172.21.15.92:0/4149997583 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812502108 0 0) 0x7f18c80c6d00 con 0x7f18c806d160 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18c7400640 1 -- 172.21.15.92:0/4149997583 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003520 con 0x7f18c806d160 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 -- 172.21.15.92:0/4149997583 >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 msgr2=0x7f18c815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 --2- 172.21.15.92:0/4149997583 >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 0x7f18c815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 -- 172.21.15.92:0/4149997583 >> v1:172.21.15.159:6789/0 conn(0x7f18c806d160 legacy=0x7f18c8152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18bf400640 1 -- 172.21.15.92:0/4149997583 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c80c6fa0 con 0x7f18c80c5f50 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.117+0000 7f18c7e00640 1 --2- 172.21.15.92:0/4149997583 >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 0x7f18c815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.118+0000 7f18bf400640 1 --2- 172.21.15.92:0/4149997583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009870 tx=0x7f18b002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90217ca0cc76c8ed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.118+0000 7f18c7400640 1 -- 172.21.15.92:0/4149997583 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002f830 con 0x7f18c80c5f50 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.118+0000 7f18c7400640 1 -- 172.21.15.92:0/4149997583 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b002f9d0 con 0x7f18c80c5f50 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.118+0000 7f18c7400640 1 -- 172.21.15.92:0/4149997583 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002fd80 con 0x7f18c80c5f50 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 -- 172.21.15.92:0/4149997583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 msgr2=0x7f18c8152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 --2- 172.21.15.92:0/4149997583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009870 tx=0x7f18b002ba10 comp rx=0 tx=0).stop 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 -- 172.21.15.92:0/4149997583 shutdown_connections 2024-03-19T03:05:14.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 --2- 172.21.15.92:0/4149997583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5f50 0x7f18c8152520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 --2- 172.21.15.92:0/4149997583 >> v2:172.21.15.159:3300/0 conn(0x7f18c80c5b70 0x7f18c815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 -- 172.21.15.92:0/4149997583 >> 172.21.15.92:0/4149997583 conn(0x7f18c80cc1f0 msgr2=0x7f18c80cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:14.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 -- 172.21.15.92:0/4149997583 shutdown_connections 2024-03-19T03:05:14.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.119+0000 7f18cda00640 1 -- 172.21.15.92:0/4149997583 wait complete. 2024-03-19T03:05:14.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.121+0000 7f18cda00640 1 Processor -- start 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.121+0000 7f18cda00640 1 -- start start 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 0x7f18c8143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c5f50 0x7f18c8157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18c80729b0 con 0x7f18c806d160 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f18c806dbb0 con 0x7f18c80c5f50 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cda00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18c806e3b0 con 0x7f18c80c5b70 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18c7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18bf400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 0x7f18c8143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18c7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36322/0 (socket says 172.21.15.92:36322) 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18c7e00640 1 -- 172.21.15.92:0/1694155991 learned_addr learned my addr 172.21.15.92:0/1694155991 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.122+0000 7f18cd000640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c5f50 0x7f18c8157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 -- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c5f50 msgr2=0x7f18c8157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c5f50 0x7f18c8157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 -- 172.21.15.92:0/1694155991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 msgr2=0x7f18c81eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 -- 172.21.15.92:0/1694155991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c81edf80 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18c7e00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18bf400640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 0x7f18c8143d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18b0033000 tx=0x7f18b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b003c040 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b0047070 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0037c70 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c8157f90 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.123+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81432b0 con 0x7f18c80c5b70 2024-03-19T03:05:14.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.124+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c80c81a0 con 0x7f18c80c5b70 2024-03-19T03:05:14.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.125+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f18b004c460 con 0x7f18c80c5b70 2024-03-19T03:05:14.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.126+0000 7f18c4c00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 0x7f18a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:14.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.126+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f18b00ad4b0 con 0x7f18c80c5b70 2024-03-19T03:05:14.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.127+0000 7f18c7e00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 0x7f18a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:14.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.127+0000 7f18c7e00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 0x7f18a406e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009560 tx=0x7f18b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:14.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.131+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f18c80c81a0 con 0x7f18c80c5b70 2024-03-19T03:05:14.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.372+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18c80c3250 con 0x7f18c80c5b70 2024-03-19T03:05:14.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.373+0000 7f18c4c00640 1 -- 172.21.15.92:0/1694155991 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6003 (secure 0 0 0) 0x7f18b0033030 con 0x7f18c80c5b70 2024-03-19T03:05:14.375 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:14.375 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:09.726694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:14.375 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 32 2024-03-19T03:05:14.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.375+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 msgr2=0x7f18a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.377 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.375+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 0x7f18a406e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009560 tx=0x7f18b8009190 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.375+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 msgr2=0x7f18c8143d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.375+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 0x7f18c8143d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18b0033000 tx=0x7f18b00046c0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 shutdown_connections 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18a406bc00 0x7f18a406e0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c5f50 0x7f18c8157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c80c5b70 0x7f18c8143d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 --2- 172.21.15.92:0/1694155991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c806d160 0x7f18c81eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 >> 172.21.15.92:0/1694155991 conn(0x7f18c80cc1f0 msgr2=0x7f18c80caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 shutdown_connections 2024-03-19T03:05:14.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:14.376+0000 7f18cda00640 1 -- 172.21.15.92:0/1694155991 wait complete. 2024-03-19T03:05:14.396 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:16.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:16.615 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.614+0000 7f0436800640 1 Processor -- start 2024-03-19T03:05:16.616 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.614+0000 7f0436800640 1 -- start start 2024-03-19T03:05:16.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.615+0000 7f0436800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f043006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.615+0000 7f0436800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f04301593d0 0x7f04300c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.617 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.615+0000 7f0436800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04300c9a80 con 0x7f04300c6470 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.615+0000 7f0436800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f04300c6c90 con 0x7f04301593d0 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.615+0000 7f0436800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0430159910 con 0x7f0430152bf0 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f04300c6470 0x7f04300c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36940/0 (socket says 172.21.15.92:36940) 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042ea00640 1 -- 172.21.15.92:0/93090951 learned_addr learned my addr 172.21.15.92:0/93090951 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042f400640 1 --2- 172.21.15.92:0/93090951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f043006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042fe00640 1 --2- 172.21.15.92:0/93090951 >> v2:172.21.15.159:3300/0 conn(0x7f04301593d0 0x7f04300c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.618 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042f400640 1 -- 172.21.15.92:0/93090951 >> v2:172.21.15.159:3300/0 conn(0x7f04301593d0 msgr2=0x7f04300c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042e000640 1 -- 172.21.15.92:0/93090951 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1397359690 0 0) 0x7f04300c9a80 con 0x7f04300c6470 2024-03-19T03:05:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042f400640 1 --2- 172.21.15.92:0/93090951 >> v2:172.21.15.159:3300/0 conn(0x7f04301593d0 0x7f04300c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042f400640 1 -- 172.21.15.92:0/93090951 >> v1:172.21.15.92:6789/0 conn(0x7f04300c6470 legacy=0x7f04300c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.616+0000 7f042f400640 1 -- 172.21.15.92:0/93090951 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04300c8650 con 0x7f0430152bf0 2024-03-19T03:05:16.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.618+0000 7f042f400640 1 --2- 172.21.15.92:0/93090951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f043006eef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097a0 tx=0x7f041c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801eca43ca6950f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:16.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.618+0000 7f042e000640 1 -- 172.21.15.92:0/93090951 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c0315a0 con 0x7f0430152bf0 2024-03-19T03:05:16.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f042e000640 1 -- 172.21.15.92:0/93090951 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041c037070 con 0x7f0430152bf0 2024-03-19T03:05:16.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f042e000640 1 -- 172.21.15.92:0/93090951 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c031960 con 0x7f0430152bf0 2024-03-19T03:05:16.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 -- 172.21.15.92:0/93090951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 msgr2=0x7f043006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 --2- 172.21.15.92:0/93090951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f043006eef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097a0 tx=0x7f041c02b840 comp rx=0 tx=0).stop 2024-03-19T03:05:16.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 -- 172.21.15.92:0/93090951 shutdown_connections 2024-03-19T03:05:16.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 --2- 172.21.15.92:0/93090951 >> v2:172.21.15.159:3300/0 conn(0x7f04301593d0 0x7f04300c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 --2- 172.21.15.92:0/93090951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f043006eef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 -- 172.21.15.92:0/93090951 >> 172.21.15.92:0/93090951 conn(0x7f04300c8d90 msgr2=0x7f04300c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:16.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 -- 172.21.15.92:0/93090951 shutdown_connections 2024-03-19T03:05:16.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.619+0000 7f0436800640 1 -- 172.21.15.92:0/93090951 wait complete. 2024-03-19T03:05:16.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.621+0000 7f0436800640 1 Processor -- start 2024-03-19T03:05:16.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.621+0000 7f0436800640 1 -- start start 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f04301f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04301593d0 0x7f0430148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f043006ea60 con 0x7f04301593d0 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f043006e4a0 con 0x7f04300c6470 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f0436800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f043006e6f0 con 0x7f0430152bf0 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.622+0000 7f042f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:50194/0 (socket says 172.21.15.92:50194) 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 -- 172.21.15.92:0/275963303 learned_addr learned my addr 172.21.15.92:0/275963303 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042ea00640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f04301f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042fe00640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04301593d0 0x7f0430148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 -- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 msgr2=0x7f04301f8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f04301f8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 -- 172.21.15.92:0/275963303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04301593d0 msgr2=0x7f0430148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04301593d0 0x7f0430148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 -- 172.21.15.92:0/275963303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04300713c0 con 0x7f04300c6470 2024-03-19T03:05:16.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.623+0000 7f042f400640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097a0 tx=0x7f041c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.624+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c004230 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.624+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041c0043f0 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.624+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c003850 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.624+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0430071660 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.624+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04300ca420 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.625+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043014cc70 con 0x7f04300c6470 2024-03-19T03:05:16.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.626+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f041c0039f0 con 0x7f04300c6470 2024-03-19T03:05:16.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.627+0000 7f0427400640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 0x7f041006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:16.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.627+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f041c02f030 con 0x7f04300c6470 2024-03-19T03:05:16.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.627+0000 7f042ea00640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 0x7f041006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:16.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.628+0000 7f042ea00640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 0x7f041006e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0430142360 tx=0x7f0420007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:16.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.632+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f043014cc70 con 0x7f04300c6470 2024-03-19T03:05:16.870 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.869+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04300684c0 con 0x7f04300c6470 2024-03-19T03:05:16.871 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.869+0000 7f0427400640 1 -- 172.21.15.92:0/275963303 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6003 (secure 0 0 0) 0x7f041c041070 con 0x7f04300c6470 2024-03-19T03:05:16.871 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:16.871 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:09.726694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:16.872 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 32 2024-03-19T03:05:16.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 msgr2=0x7f041006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 0x7f041006e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0430142360 tx=0x7f0420007d50 comp rx=0 tx=0).stop 2024-03-19T03:05:16.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 msgr2=0x7f0430155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:16.874 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097a0 tx=0x7f041c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 shutdown_connections 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f041006bbd0 0x7f041006e090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04301593d0 0x7f0430148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0430152bf0 0x7f04301f8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 --2- 172.21.15.92:0/275963303 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04300c6470 0x7f0430155d10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.872+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 >> 172.21.15.92:0/275963303 conn(0x7f04300c8d90 msgr2=0x7f043014e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.873+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 shutdown_connections 2024-03-19T03:05:16.875 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:16.873+0000 7f0436800640 1 -- 172.21.15.92:0/275963303 wait complete. 2024-03-19T03:05:16.894 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:18.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.098+0000 7f18d0800640 1 Processor -- start 2024-03-19T03:05:19.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.098+0000 7f18d0800640 1 -- start start 2024-03-19T03:05:19.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 0x7f18c8158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 0x7f18c80723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 0x7f18c80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f18c8158860 con 0x7f18c80c7da0 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f18c80c3b80 con 0x7f18c8151ce0 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.099+0000 7f18d0800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18c8072e70 con 0x7f18c8151900 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18ce800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 0x7f18c80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 0x7f18c8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cd400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 0x7f18c80723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18ce800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 0x7f18c80c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53734/0 (socket says 172.21.15.92:53734) 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18ce800640 1 -- 172.21.15.92:0/300388761 learned_addr learned my addr 172.21.15.92:0/300388761 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 -- 172.21.15.92:0/300388761 >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 msgr2=0x7f18c80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 --2- 172.21.15.92:0/300388761 >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 0x7f18c80723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 -- 172.21.15.92:0/300388761 >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 msgr2=0x7f18c80c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:19.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 --2- 172.21.15.92:0/300388761 >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 0x7f18c80c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cde00640 1 -- 172.21.15.92:0/300388761 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c8073150 con 0x7f18c8151900 2024-03-19T03:05:19.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.100+0000 7f18cd400640 1 --2- 172.21.15.92:0/300388761 >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 0x7f18c80723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.101+0000 7f18cde00640 1 --2- 172.21.15.92:0/300388761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 0x7f18c8158320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f18c8074fd0 tx=0x7f18bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8335a112deef58fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18cca00640 1 -- 172.21.15.92:0/300388761 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc035020 con 0x7f18c8151900 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18cca00640 1 -- 172.21.15.92:0/300388761 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc031830 con 0x7f18c8151900 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18cca00640 1 -- 172.21.15.92:0/300388761 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc031c00 con 0x7f18c8151900 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 -- 172.21.15.92:0/300388761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 msgr2=0x7f18c8158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 --2- 172.21.15.92:0/300388761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 0x7f18c8158320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f18c8074fd0 tx=0x7f18bc02b8d0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 -- 172.21.15.92:0/300388761 shutdown_connections 2024-03-19T03:05:19.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 --2- 172.21.15.92:0/300388761 >> v2:172.21.15.92:3300/0 conn(0x7f18c80c7da0 0x7f18c80c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 --2- 172.21.15.92:0/300388761 >> v2:172.21.15.159:3300/0 conn(0x7f18c8151ce0 0x7f18c80723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 --2- 172.21.15.92:0/300388761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151900 0x7f18c8158320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 -- 172.21.15.92:0/300388761 >> 172.21.15.92:0/300388761 conn(0x7f18c80c8d90 msgr2=0x7f18c80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 -- 172.21.15.92:0/300388761 shutdown_connections 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.102+0000 7f18d0800640 1 -- 172.21.15.92:0/300388761 wait complete. 2024-03-19T03:05:19.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.104+0000 7f18d0800640 1 Processor -- start 2024-03-19T03:05:19.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.104+0000 7f18d0800640 1 -- start start 2024-03-19T03:05:19.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151ce0 0x7f18c814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8072910 con 0x7f18c8151900 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8071eb0 con 0x7f18c80c7da0 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18d0800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18c80c9af0 con 0x7f18c8151ce0 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18cd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18cde00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18cde00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:50198/0 (socket says 172.21.15.92:50198) 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18cde00640 1 -- 172.21.15.92:0/1538248218 learned_addr learned my addr 172.21.15.92:0/1538248218 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18cd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53750/0 (socket says 172.21.15.92:53750) 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.105+0000 7f18ce800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151ce0 0x7f18c814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 -- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151ce0 msgr2=0x7f18c814f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151ce0 0x7f18c814f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 -- 172.21.15.92:0/1538248218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 msgr2=0x7f18c814f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 -- 172.21.15.92:0/1538248218 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c80c7130 con 0x7f18c80c7da0 2024-03-19T03:05:19.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cd400640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:19.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18cde00640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000c00 tx=0x7f18bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:19.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc040070 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc03a850 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c81e69d0 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.106+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc035040 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.107+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c80c73d0 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.108+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c80684c0 con 0x7f18c80c7da0 2024-03-19T03:05:19.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.109+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f18bc038020 con 0x7f18c80c7da0 2024-03-19T03:05:19.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.109+0000 7f18c2000640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 0x7f18ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:19.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.110+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f18bc0abd20 con 0x7f18c80c7da0 2024-03-19T03:05:19.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.110+0000 7f18cd400640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 0x7f18ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:19.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.110+0000 7f18cd400640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 0x7f18ac06e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18c80c7c70 tx=0x7f18b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:19.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.114+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f18bc079e50 con 0x7f18c80c7da0 2024-03-19T03:05:19.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.356+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18c8071600 con 0x7f18c80c7da0 2024-03-19T03:05:19.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.357+0000 7f18c2000640 1 -- 172.21.15.92:0/1538248218 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6003 (secure 0 0 0) 0x7f18bc07def0 con 0x7f18c80c7da0 2024-03-19T03:05:19.359 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:19.359 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:09.726694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:19.359 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 32 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.359+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 msgr2=0x7f18ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.359+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 0x7f18ac06e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18c80c7c70 tx=0x7f18b8009190 comp rx=0 tx=0).stop 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 msgr2=0x7f18c814cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000c00 tx=0x7f18bc0046c0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 shutdown_connections 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f18ac06bbd0 0x7f18ac06e090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.362 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c8151ce0 0x7f18c814f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18c8151900 0x7f18c814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 --2- 172.21.15.92:0/1538248218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c80c7da0 0x7f18c814cec0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:19.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 >> 172.21.15.92:0/1538248218 conn(0x7f18c80c8d90 msgr2=0x7f18c80c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:19.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 shutdown_connections 2024-03-19T03:05:19.363 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:19.360+0000 7f18d0800640 1 -- 172.21.15.92:0/1538248218 wait complete. 2024-03-19T03:05:19.379 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:21.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:21.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.596+0000 7fddc9e00640 1 Processor -- start 2024-03-19T03:05:21.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.596+0000 7fddc9e00640 1 -- start start 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.597+0000 7fddc9e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fddc4152bf0 0x7fddc406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.597+0000 7fddc9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fddc40c6470 0x7fddc40c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.597+0000 7fddc9e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddc40c9a80 con 0x7fddc41593d0 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.597+0000 7fddc9e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fddc40c6c90 con 0x7fddc40c6470 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.597+0000 7fddc9e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fddc4159910 con 0x7fddc4152bf0 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fddc4152bf0 0x7fddc406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc3400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fddc41593d0 0x7fddc40c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36948/0 (socket says 172.21.15.92:36948) 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc3400640 1 -- 172.21.15.92:0/1581761846 learned_addr learned my addr 172.21.15.92:0/1581761846 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2000640 1 --2- 172.21.15.92:0/1581761846 >> v2:172.21.15.92:3300/0 conn(0x7fddc40c6470 0x7fddc40c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2a00640 1 -- 172.21.15.92:0/1581761846 >> v2:172.21.15.92:3300/0 conn(0x7fddc40c6470 msgr2=0x7fddc40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2a00640 1 --2- 172.21.15.92:0/1581761846 >> v2:172.21.15.92:3300/0 conn(0x7fddc40c6470 0x7fddc40c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2a00640 1 -- 172.21.15.92:0/1581761846 >> v1:172.21.15.92:6789/0 conn(0x7fddc41593d0 legacy=0x7fddc40c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc1600640 1 -- 172.21.15.92:0/1581761846 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 926966653 0 0) 0x7fddc40c9a80 con 0x7fddc41593d0 2024-03-19T03:05:21.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.598+0000 7fddc2a00640 1 -- 172.21.15.92:0/1581761846 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc40c8650 con 0x7fddc4152bf0 2024-03-19T03:05:21.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.600+0000 7fddc2a00640 1 --2- 172.21.15.92:0/1581761846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc4152bf0 0x7fddc406eef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fddc4075180 tx=0x7fddb402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc005d79a7f3935d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:21.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.600+0000 7fddc1600640 1 -- 172.21.15.92:0/1581761846 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb40315a0 con 0x7fddc4152bf0 2024-03-19T03:05:21.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.600+0000 7fddc1600640 1 -- 172.21.15.92:0/1581761846 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb4037070 con 0x7fddc4152bf0 2024-03-19T03:05:21.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.600+0000 7fddc1600640 1 -- 172.21.15.92:0/1581761846 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4031960 con 0x7fddc4152bf0 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 -- 172.21.15.92:0/1581761846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc4152bf0 msgr2=0x7fddc406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1581761846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc4152bf0 0x7fddc406eef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fddc4075180 tx=0x7fddb402b840 comp rx=0 tx=0).stop 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 -- 172.21.15.92:0/1581761846 shutdown_connections 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1581761846 >> v2:172.21.15.92:3300/0 conn(0x7fddc40c6470 0x7fddc40c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1581761846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc4152bf0 0x7fddc406eef0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 -- 172.21.15.92:0/1581761846 >> 172.21.15.92:0/1581761846 conn(0x7fddc40c8d90 msgr2=0x7fddc40c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 -- 172.21.15.92:0/1581761846 shutdown_connections 2024-03-19T03:05:21.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.601+0000 7fddc9e00640 1 -- 172.21.15.92:0/1581761846 wait complete. 2024-03-19T03:05:21.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.602+0000 7fddc9e00640 1 Processor -- start 2024-03-19T03:05:21.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 -- start start 2024-03-19T03:05:21.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc40c6470 0x7fddc41f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddc41593d0 0x7fddc4144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fddc406ea60 con 0x7fddc4152bf0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fddc406e4a0 con 0x7fddc40c6470 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.603+0000 7fddc9e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fddc406e6f0 con 0x7fddc41593d0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53766/0 (socket says 172.21.15.92:53766) 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 -- 172.21.15.92:0/1512185024 learned_addr learned my addr 172.21.15.92:0/1512185024 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2a00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc40c6470 0x7fddc41f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc3400640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddc41593d0 0x7fddc4144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 -- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddc41593d0 msgr2=0x7fddc4144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddc41593d0 0x7fddc4144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 -- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc40c6470 msgr2=0x7fddc41f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc40c6470 0x7fddc41f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 -- 172.21.15.92:0/1512185024 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc4072f90 con 0x7fddc4152bf0 2024-03-19T03:05:21.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.604+0000 7fddc2000640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fddb000e9b0 tx=0x7fddb000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.605+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb0017070 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.605+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb000c9a0 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.605+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb000cd90 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.605+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc4143890 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.605+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc4143e50 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.606+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc40684c0 con 0x7fddc4152bf0 2024-03-19T03:05:21.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.607+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fddb001a070 con 0x7fddc4152bf0 2024-03-19T03:05:21.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.608+0000 7fddbb400640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 0x7fdda406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:21.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.608+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fddb008afa0 con 0x7fddc4152bf0 2024-03-19T03:05:21.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.608+0000 7fddc2a00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 0x7fdda406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:21.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.609+0000 7fddc2a00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 0x7fdda406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fddb4036000 tx=0x7fddb4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:21.614 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.613+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fddb00591f0 con 0x7fddc4152bf0 2024-03-19T03:05:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.852+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddc4148c00 con 0x7fddc4152bf0 2024-03-19T03:05:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.853+0000 7fddbb400640 1 -- 172.21.15.92:0/1512185024 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6007 (secure 0 0 0) 0x7fddb005f030 con 0x7fddc4152bf0 2024-03-19T03:05:21.855 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:21.855 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:18.527534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 33 2024-03-19T03:05:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 msgr2=0x7fdda406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 0x7fdda406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fddb4036000 tx=0x7fddb4008030 comp rx=0 tx=0).stop 2024-03-19T03:05:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 msgr2=0x7fddc4072770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fddb000e9b0 tx=0x7fddb000ee70 comp rx=0 tx=0).stop 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 shutdown_connections 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdda406bbd0 0x7fdda406e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fddc41593d0 0x7fddc4144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc4152bf0 0x7fddc4072770 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 --2- 172.21.15.92:0/1512185024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fddc40c6470 0x7fddc41f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 >> 172.21.15.92:0/1512185024 conn(0x7fddc40c8d90 msgr2=0x7fddc4150410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 shutdown_connections 2024-03-19T03:05:21.859 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:21.856+0000 7fddc9e00640 1 -- 172.21.15.92:0/1512185024 wait complete. 2024-03-19T03:05:21.876 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:23.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:24.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.094+0000 7f87a2800640 1 Processor -- start 2024-03-19T03:05:24.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.094+0000 7f87a2800640 1 -- start start 2024-03-19T03:05:24.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 0x7f879c152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f879c06d160 0x7f879c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f879c06df20 con 0x7f879c0c5f50 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f879c152fa0 con 0x7f879c06d160 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a2800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f879c0c6d00 con 0x7f879c0c5b70 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a1e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f879c06d160 0x7f879c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a0a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 0x7f879c152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a1e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f879c06d160 0x7f879c152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:50230/0 (socket says 172.21.15.92:50230) 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.096+0000 7f87a1e00640 1 -- 172.21.15.92:0/1397539560 learned_addr learned my addr 172.21.15.92:0/1397539560 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1e00640 1 -- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 msgr2=0x7f879c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1e00640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1e00640 1 -- 172.21.15.92:0/1397539560 >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 msgr2=0x7f879c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1e00640 1 --2- 172.21.15.92:0/1397539560 >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 0x7f879c152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1e00640 1 -- 172.21.15.92:0/1397539560 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c06d610 con 0x7f879c06d160 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a1400640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:24.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.097+0000 7f87a0a00640 1 --2- 172.21.15.92:0/1397539560 >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 0x7f879c152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.098+0000 7f87a1e00640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c06d160 0x7f879c152a60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f879c0c6870 tx=0x7f879402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a57899580b57125c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.098+0000 7f879be00640 1 -- 172.21.15.92:0/1397539560 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879402cb00 con 0x7f879c06d160 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.098+0000 7f879be00640 1 -- 172.21.15.92:0/1397539560 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879402cca0 con 0x7f879c06d160 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.098+0000 7f879be00640 1 -- 172.21.15.92:0/1397539560 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8794034610 con 0x7f879c06d160 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 -- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c06d160 msgr2=0x7f879c152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c06d160 0x7f879c152a60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f879c0c6870 tx=0x7f879402bf60 comp rx=0 tx=0).stop 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 -- 172.21.15.92:0/1397539560 shutdown_connections 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c06d160 0x7f879c152a60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 --2- 172.21.15.92:0/1397539560 >> v2:172.21.15.92:3300/0 conn(0x7f879c0c5f50 0x7f879c152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 --2- 172.21.15.92:0/1397539560 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 -- 172.21.15.92:0/1397539560 >> 172.21.15.92:0/1397539560 conn(0x7f879c0cc1f0 msgr2=0x7f879c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:24.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 -- 172.21.15.92:0/1397539560 shutdown_connections 2024-03-19T03:05:24.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.099+0000 7f87a2800640 1 -- 172.21.15.92:0/1397539560 wait complete. 2024-03-19T03:05:24.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.101+0000 7f87a2800640 1 Processor -- start 2024-03-19T03:05:24.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.101+0000 7f87a2800640 1 -- start start 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c5f50 0x7f879c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f879c06dbb0 con 0x7f879c06d160 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f879c06d8d0 con 0x7f879c0c5f50 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a2800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f879c0729b0 con 0x7f879c0c5b70 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53782/0 (socket says 172.21.15.92:53782) 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 -- 172.21.15.92:0/207612693 learned_addr learned my addr 172.21.15.92:0/207612693 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:24.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a0a00640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1e00640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c5f50 0x7f879c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 -- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 msgr2=0x7f879c146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 -- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c5f50 msgr2=0x7f879c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c5f50 0x7f879c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a1400640 1 -- 172.21.15.92:0/207612693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c1f2640 con 0x7f879c06d160 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.102+0000 7f87a0a00640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:05:24.105 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f87a1400640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f879000e980 tx=0x7f879000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790017070 con 0x7f879c06d160 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879000ca00 con 0x7f879c06d160 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879000cdd0 con 0x7f879c06d160 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879c1426e0 con 0x7f879c06d160 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.103+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879c142c60 con 0x7f879c06d160 2024-03-19T03:05:24.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.104+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f879c04f580 con 0x7f879c06d160 2024-03-19T03:05:24.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.105+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f879001a070 con 0x7f879c06d160 2024-03-19T03:05:24.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.106+0000 7f8799600640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 0x7f877c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:24.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.106+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f879008acd0 con 0x7f879c06d160 2024-03-19T03:05:24.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.106+0000 7f87a0a00640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 0x7f877c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:24.108 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.107+0000 7f87a0a00640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 0x7f877c06e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f879c15cf50 tx=0x7f8788008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:24.112 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.110+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8790059450 con 0x7f879c06d160 2024-03-19T03:05:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.347+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f879c152b80 con 0x7f879c06d160 2024-03-19T03:05:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.348+0000 7f8799600640 1 -- 172.21.15.92:0/207612693 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6007 (secure 0 0 0) 0x7f8790058d50 con 0x7f879c06d160 2024-03-19T03:05:24.350 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:24.350 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:18.527534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:24.351 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 33 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 msgr2=0x7f877c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 0x7f877c06e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f879c15cf50 tx=0x7f8788008040 comp rx=0 tx=0).stop 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 msgr2=0x7f879c1f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f879000e980 tx=0x7f879000ee40 comp rx=0 tx=0).stop 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 shutdown_connections 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bbd0 0x7f877c06e090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c5f50 0x7f879c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c0c5b70 0x7f879c146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 --2- 172.21.15.92:0/207612693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c06d160 0x7f879c1f2100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 >> 172.21.15.92:0/207612693 conn(0x7f879c0cc1f0 msgr2=0x7f879c0c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 shutdown_connections 2024-03-19T03:05:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:24.351+0000 7f87a2800640 1 -- 172.21.15.92:0/207612693 wait complete. 2024-03-19T03:05:24.370 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5347": { "addr": "172.21.15.159:6837/3372688839", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 3372688839, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 3372688839, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5347, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:05:18.527534+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5347 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:05:26.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:26.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.590+0000 7f80bca00640 1 Processor -- start 2024-03-19T03:05:26.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.590+0000 7f80bca00640 1 -- start start 2024-03-19T03:05:26.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80bca00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80bca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80bca00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80b80c3b80 con 0x7f80b8151900 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80bca00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f80b8158860 con 0x7f80b8151ce0 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80bca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f80b8072e70 con 0x7f80b80c7da0 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80b5400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80b6800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80b5400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39396/0 (socket says 172.21.15.92:39396) 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.592+0000 7f80b5400640 1 -- 172.21.15.92:0/1727062117 learned_addr learned my addr 172.21.15.92:0/1727062117 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b6800640 1 -- 172.21.15.92:0/1727062117 >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 msgr2=0x7f80b8158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b6800640 1 --2- 172.21.15.92:0/1727062117 >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b6800640 1 -- 172.21.15.92:0/1727062117 >> v1:172.21.15.159:6789/0 conn(0x7f80b8151900 legacy=0x7f80b80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b6800640 1 -- 172.21.15.92:0/1727062117 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b8073150 con 0x7f80b80c7da0 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b4a00640 1 -- 172.21.15.92:0/1727062117 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288678031 0 0) 0x7f80b80c3b80 con 0x7f80b8151900 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.593+0000 7f80b5400640 1 --2- 172.21.15.92:0/1727062117 >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.594+0000 7f80b6800640 1 --2- 172.21.15.92:0/1727062117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b80c8230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009d90 tx=0x7f80b002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e58dc9d9e95b4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.594+0000 7f80b4a00640 1 -- 172.21.15.92:0/1727062117 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0039070 con 0x7f80b80c7da0 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.594+0000 7f80b4a00640 1 -- 172.21.15.92:0/1727062117 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b002cce0 con 0x7f80b80c7da0 2024-03-19T03:05:26.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.594+0000 7f80b4a00640 1 -- 172.21.15.92:0/1727062117 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0034660 con 0x7f80b80c7da0 2024-03-19T03:05:26.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 -- 172.21.15.92:0/1727062117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 msgr2=0x7f80b80c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 --2- 172.21.15.92:0/1727062117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b80c8230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009d90 tx=0x7f80b002bf30 comp rx=0 tx=0).stop 2024-03-19T03:05:26.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 -- 172.21.15.92:0/1727062117 shutdown_connections 2024-03-19T03:05:26.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 --2- 172.21.15.92:0/1727062117 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b80c8230 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 --2- 172.21.15.92:0/1727062117 >> v2:172.21.15.159:3300/0 conn(0x7f80b8151ce0 0x7f80b8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 -- 172.21.15.92:0/1727062117 >> 172.21.15.92:0/1727062117 conn(0x7f80b80c8d90 msgr2=0x7f80b80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:26.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 -- 172.21.15.92:0/1727062117 shutdown_connections 2024-03-19T03:05:26.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.595+0000 7f80bca00640 1 -- 172.21.15.92:0/1727062117 wait complete. 2024-03-19T03:05:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.596+0000 7f80bca00640 1 Processor -- start 2024-03-19T03:05:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.597+0000 7f80bca00640 1 -- start start 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b8156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80b8151ce0 0x7f80b8148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f80b80c9af0 con 0x7f80b8151900 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f80b8071eb0 con 0x7f80b8151ce0 2024-03-19T03:05:26.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80bca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f80b80c9c70 con 0x7f80b80c7da0 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33064/0 (socket says 172.21.15.92:33064) 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 -- 172.21.15.92:0/1836980123 learned_addr learned my addr 172.21.15.92:0/1836980123 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5e00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b8156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b6800640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80b8151ce0 0x7f80b8148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 -- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 msgr2=0x7f80b8156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b8156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 -- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80b8151ce0 msgr2=0x7f80b8148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80b8151ce0 0x7f80b8148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.598+0000 7f80b5400640 1 -- 172.21.15.92:0/1836980123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b814c720 con 0x7f80b8151900 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80b5400640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80a400ed70 tx=0x7f80a400d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4010850 con 0x7f80b8151900 2024-03-19T03:05:26.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a40109f0 con 0x7f80b8151900 2024-03-19T03:05:26.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4010da0 con 0x7f80b8151900 2024-03-19T03:05:26.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b814c9e0 con 0x7f80b8151900 2024-03-19T03:05:26.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.599+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b81f37e0 con 0x7f80b8151900 2024-03-19T03:05:26.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.600+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b814f310 con 0x7f80b8151900 2024-03-19T03:05:26.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.601+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f80a400f3d0 con 0x7f80b8151900 2024-03-19T03:05:26.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.602+0000 7f80ae000640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 0x7f809806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:26.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.602+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f80a408b380 con 0x7f80b8151900 2024-03-19T03:05:26.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.602+0000 7f80b5e00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 0x7f809806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:26.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.603+0000 7f80b5e00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 0x7f809806e0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f80a8000f60 tx=0x7f80a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:26.611 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.606+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f80b814f310 con 0x7f80b8151900 2024-03-19T03:05:26.837 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.836+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80b80684c0 con 0x7f80b8151900 2024-03-19T03:05:26.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.837+0000 7f80ae000640 1 -- 172.21.15.92:0/1836980123 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6004 (secure 0 0 0) 0x7f80a40595d0 con 0x7f80b8151900 2024-03-19T03:05:26.839 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:26.839 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:24.477962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:26.839 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 34 2024-03-19T03:05:26.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 msgr2=0x7f809806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 0x7f809806e0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f80a8000f60 tx=0x7f80a8009040 comp rx=0 tx=0).stop 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 msgr2=0x7f80b81f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80a400ed70 tx=0x7f80a400d720 comp rx=0 tx=0).stop 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 shutdown_connections 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f809806bc00 0x7f809806e0c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80b8151ce0 0x7f80b8148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b8151900 0x7f80b81f3fe0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 --2- 172.21.15.92:0/1836980123 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80b80c7da0 0x7f80b8156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:26.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 >> 172.21.15.92:0/1836980123 conn(0x7f80b80c8d90 msgr2=0x7f80b806ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:26.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.839+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 shutdown_connections 2024-03-19T03:05:26.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:26.840+0000 7f80bca00640 1 -- 172.21.15.92:0/1836980123 wait complete. 2024-03-19T03:05:26.859 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:28.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:29.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.080+0000 7f3491600640 1 Processor -- start 2024-03-19T03:05:29.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.080+0000 7f3491600640 1 -- start start 2024-03-19T03:05:29.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3491600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 0x7f348c0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3491600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f348c1593d0 0x7f348c0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3491600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f348c0c9a80 con 0x7f348c152bf0 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3491600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f348c0c6c90 con 0x7f348c1593d0 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3491600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f348c159910 con 0x7f348c0c6470 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f3483e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 0x7f348c0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f348aa00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f348c152bf0 0x7f348c06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:47294/0 (socket says 172.21.15.92:47294) 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f348aa00640 1 -- 172.21.15.92:0/2081381015 learned_addr learned my addr 172.21.15.92:0/2081381015 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:29.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.082+0000 7f348b400640 1 --2- 172.21.15.92:0/2081381015 >> v2:172.21.15.92:3300/0 conn(0x7f348c1593d0 0x7f348c0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.083+0000 7f3483e00640 1 -- 172.21.15.92:0/2081381015 >> v2:172.21.15.92:3300/0 conn(0x7f348c1593d0 msgr2=0x7f348c0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:29.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.083+0000 7f3483e00640 1 --2- 172.21.15.92:0/2081381015 >> v2:172.21.15.92:3300/0 conn(0x7f348c1593d0 0x7f348c0c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.083+0000 7f3483e00640 1 -- 172.21.15.92:0/2081381015 >> v1:172.21.15.159:6789/0 conn(0x7f348c152bf0 legacy=0x7f348c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.083+0000 7f348a000640 1 -- 172.21.15.92:0/2081381015 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1422221230 0 0) 0x7f348c0c9a80 con 0x7f348c152bf0 2024-03-19T03:05:29.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.083+0000 7f3483e00640 1 -- 172.21.15.92:0/2081381015 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348c0c8650 con 0x7f348c0c6470 2024-03-19T03:05:29.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.084+0000 7f3483e00640 1 --2- 172.21.15.92:0/2081381015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 0x7f348c0c3f80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f34740097a0 tx=0x7f347402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e40cfdf2931812 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:29.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.084+0000 7f348a000640 1 -- 172.21.15.92:0/2081381015 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474031780 con 0x7f348c0c6470 2024-03-19T03:05:29.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.084+0000 7f348a000640 1 -- 172.21.15.92:0/2081381015 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3474037070 con 0x7f348c0c6470 2024-03-19T03:05:29.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.084+0000 7f348a000640 1 -- 172.21.15.92:0/2081381015 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474031b40 con 0x7f348c0c6470 2024-03-19T03:05:29.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 -- 172.21.15.92:0/2081381015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 msgr2=0x7f348c0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 --2- 172.21.15.92:0/2081381015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 0x7f348c0c3f80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f34740097a0 tx=0x7f347402b820 comp rx=0 tx=0).stop 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 -- 172.21.15.92:0/2081381015 shutdown_connections 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 --2- 172.21.15.92:0/2081381015 >> v2:172.21.15.92:3300/0 conn(0x7f348c1593d0 0x7f348c0c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 --2- 172.21.15.92:0/2081381015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c0c6470 0x7f348c0c3f80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 -- 172.21.15.92:0/2081381015 >> 172.21.15.92:0/2081381015 conn(0x7f348c0c8d90 msgr2=0x7f348c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 -- 172.21.15.92:0/2081381015 shutdown_connections 2024-03-19T03:05:29.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.085+0000 7f3491600640 1 -- 172.21.15.92:0/2081381015 wait complete. 2024-03-19T03:05:29.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.087+0000 7f3491600640 1 Processor -- start 2024-03-19T03:05:29.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.087+0000 7f3491600640 1 -- start start 2024-03-19T03:05:29.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f348c152bf0 0x7f348c156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c1593d0 0x7f348c1e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f348c06ea60 con 0x7f348c0c6470 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f348c06e4a0 con 0x7f348c152bf0 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3491600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f348c06e6f0 con 0x7f348c1593d0 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33076/0 (socket says 172.21.15.92:33076) 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 -- 172.21.15.92:0/2540224747 learned_addr learned my addr 172.21.15.92:0/2540224747 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f3483e00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f348c152bf0 0x7f348c156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348b400640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c1593d0 0x7f348c1e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 -- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c1593d0 msgr2=0x7f348c1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c1593d0 0x7f348c1e7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 -- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f348c152bf0 msgr2=0x7f348c156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f348c152bf0 0x7f348c156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.088+0000 7f348aa00640 1 -- 172.21.15.92:0/2540224747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348c1f3ed0 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f348aa00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f347c00c4d0 tx=0x7f347c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347c018070 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f347c007500 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347c004a20 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348c155d90 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.089+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348c1f4420 con 0x7f348c0c6470 2024-03-19T03:05:29.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.090+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f348c14eae0 con 0x7f348c0c6470 2024-03-19T03:05:29.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.091+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f347c004bc0 con 0x7f348c0c6470 2024-03-19T03:05:29.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.092+0000 7f3482a00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 0x7f346806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:29.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.092+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f347c08a000 con 0x7f348c0c6470 2024-03-19T03:05:29.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.092+0000 7f3483e00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 0x7f346806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:29.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.093+0000 7f3483e00640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 0x7f346806e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3474036000 tx=0x7f3474008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:29.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.096+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f348c14eae0 con 0x7f348c0c6470 2024-03-19T03:05:29.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.333+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f348c0684c0 con 0x7f348c0c6470 2024-03-19T03:05:29.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.334+0000 7f3482a00640 1 -- 172.21.15.92:0/2540224747 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6004 (secure 0 0 0) 0x7f347c058130 con 0x7f348c0c6470 2024-03-19T03:05:29.336 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:29.336 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:24.477962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:29.337 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 34 2024-03-19T03:05:29.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 msgr2=0x7f346806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 0x7f346806e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3474036000 tx=0x7f3474008030 comp rx=0 tx=0).stop 2024-03-19T03:05:29.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 msgr2=0x7f348c072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:29.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f347c00c4d0 tx=0x7f347c00c990 comp rx=0 tx=0).stop 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 shutdown_connections 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f346806bbd0 0x7f346806e090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f348c1593d0 0x7f348c1e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f348c152bf0 0x7f348c156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 --2- 172.21.15.92:0/2540224747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f348c0c6470 0x7f348c072bf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 >> 172.21.15.92:0/2540224747 conn(0x7f348c0c8d90 msgr2=0x7f348c06eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 shutdown_connections 2024-03-19T03:05:29.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:29.337+0000 7f3491600640 1 -- 172.21.15.92:0/2540224747 wait complete. 2024-03-19T03:05:29.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:31.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:31.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.577+0000 7fdf90800640 1 Processor -- start 2024-03-19T03:05:31.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.577+0000 7fdf90800640 1 -- start start 2024-03-19T03:05:31.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.578+0000 7fdf90800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf88151900 0x7fdf88158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.578+0000 7fdf90800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdf880c7da0 0x7fdf880c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.578+0000 7fdf90800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf880c3b80 con 0x7fdf88151ce0 2024-03-19T03:05:31.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.578+0000 7fdf90800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdf88158860 con 0x7fdf88151900 2024-03-19T03:05:31.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.578+0000 7fdf90800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fdf88072e70 con 0x7fdf880c7da0 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf88151900 0x7fdf88158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf88151900 0x7fdf88158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33080/0 (socket says 172.21.15.92:33080) 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 -- 172.21.15.92:0/721098414 learned_addr learned my addr 172.21.15.92:0/721098414 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8e800640 1 --2- 172.21.15.92:0/721098414 >> v2:172.21.15.159:3300/0 conn(0x7fdf880c7da0 0x7fdf880c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 -- 172.21.15.92:0/721098414 >> v2:172.21.15.159:3300/0 conn(0x7fdf880c7da0 msgr2=0x7fdf880c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:31.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 --2- 172.21.15.92:0/721098414 >> v2:172.21.15.159:3300/0 conn(0x7fdf880c7da0 0x7fdf880c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 -- 172.21.15.92:0/721098414 >> v1:172.21.15.159:6789/0 conn(0x7fdf88151ce0 legacy=0x7fdf880723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.579+0000 7fdf8de00640 1 -- 172.21.15.92:0/721098414 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf88073150 con 0x7fdf88151900 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.580+0000 7fdf8de00640 1 --2- 172.21.15.92:0/721098414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151900 0x7fdf88158320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdf88073700 tx=0x7fdf7c02ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e10b1fe1beacab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.580+0000 7fdf8ca00640 1 -- 172.21.15.92:0/721098414 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c02f860 con 0x7fdf88151900 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf8ca00640 1 -- 172.21.15.92:0/721098414 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7c02fa00 con 0x7fdf88151900 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf8ca00640 1 -- 172.21.15.92:0/721098414 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c02fdd0 con 0x7fdf88151900 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 -- 172.21.15.92:0/721098414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151900 msgr2=0x7fdf88158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 --2- 172.21.15.92:0/721098414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151900 0x7fdf88158320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdf88073700 tx=0x7fdf7c02ba40 comp rx=0 tx=0).stop 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 -- 172.21.15.92:0/721098414 shutdown_connections 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 --2- 172.21.15.92:0/721098414 >> v2:172.21.15.159:3300/0 conn(0x7fdf880c7da0 0x7fdf880c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 --2- 172.21.15.92:0/721098414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151900 0x7fdf88158320 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 -- 172.21.15.92:0/721098414 >> 172.21.15.92:0/721098414 conn(0x7fdf880c8d90 msgr2=0x7fdf880c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 -- 172.21.15.92:0/721098414 shutdown_connections 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.581+0000 7fdf90800640 1 -- 172.21.15.92:0/721098414 wait complete. 2024-03-19T03:05:31.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.583+0000 7fdf90800640 1 Processor -- start 2024-03-19T03:05:31.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.583+0000 7fdf90800640 1 -- start start 2024-03-19T03:05:31.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 0x7fdf8814ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 0x7fdf8814bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdf880c9af0 con 0x7fdf88151ce0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdf88071eb0 con 0x7fdf880c7da0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf90800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdf880c9c70 con 0x7fdf88151900 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf8e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf8e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33090/0 (socket says 172.21.15.92:33090) 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf8e800640 1 -- 172.21.15.92:0/3627512500 learned_addr learned my addr 172.21.15.92:0/3627512500 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf8de00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 0x7fdf8814ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.584+0000 7fdf8d400640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 0x7fdf8814bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 -- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 msgr2=0x7fdf8814bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 0x7fdf8814bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 -- 172.21.15.92:0/3627512500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 msgr2=0x7fdf88153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 -- 172.21.15.92:0/3627512500 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf88152340 con 0x7fdf880c7da0 2024-03-19T03:05:31.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8e800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8d400640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 0x7fdf8814bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf8de00640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 0x7fdf8814ce20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdf880c3d50 tx=0x7fdf7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.585+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c047070 con 0x7fdf880c7da0 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.586+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7c037d10 con 0x7fdf880c7da0 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.586+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0426c0 con 0x7fdf880c7da0 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.586+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf88152620 con 0x7fdf880c7da0 2024-03-19T03:05:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.586+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf881548e0 con 0x7fdf880c7da0 2024-03-19T03:05:31.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.587+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf880cadb0 con 0x7fdf880c7da0 2024-03-19T03:05:31.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.588+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdf7c04a070 con 0x7fdf880c7da0 2024-03-19T03:05:31.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.588+0000 7fdf82000640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:31.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.589+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fdf7c0ad060 con 0x7fdf880c7da0 2024-03-19T03:05:31.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.589+0000 7fdf8d400640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:31.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.589+0000 7fdf8d400640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf8806efa0 tx=0x7fdf78009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:31.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.593+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdf880cadb0 con 0x7fdf880c7da0 2024-03-19T03:05:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.832+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf880c4b80 con 0x7fdf880c7da0 2024-03-19T03:05:31.835 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.834+0000 7fdf82000640 1 -- 172.21.15.92:0/3627512500 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6004 (secure 0 0 0) 0x7fdf7c033030 con 0x7fdf880c7da0 2024-03-19T03:05:31.835 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:31.835 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:24.477962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:31.835 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 34 2024-03-19T03:05:31.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 msgr2=0x7fdf6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf8806efa0 tx=0x7fdf78009040 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 msgr2=0x7fdf8814ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 0x7fdf8814ce20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdf880c3d50 tx=0x7fdf7c004730 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 shutdown_connections 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf88151ce0 0x7fdf88153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf88151900 0x7fdf8814bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 --2- 172.21.15.92:0/3627512500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf880c7da0 0x7fdf8814ce20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.836+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 >> 172.21.15.92:0/3627512500 conn(0x7fdf880c8d90 msgr2=0x7fdf8806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.837+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 shutdown_connections 2024-03-19T03:05:31.839 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:31.837+0000 7fdf90800640 1 -- 172.21.15.92:0/3627512500 wait complete. 2024-03-19T03:05:31.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:32.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:05:32.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:05:33.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:34.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.073+0000 7f69b0a00640 1 Processor -- start 2024-03-19T03:05:34.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.073+0000 7f69b0a00640 1 -- start start 2024-03-19T03:05:34.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f69ac0c6470 0x7f69ac0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69ac1593d0 0x7f69ac0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f69ac0c6c90 con 0x7f69ac1593d0 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f69ac159910 con 0x7f69ac0c6470 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69b0a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f69ac06e4a0 con 0x7f69ac152bf0 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.075+0000 7f69aa800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69ac1593d0 0x7f69ac0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69ac1593d0 0x7f69ac0c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33096/0 (socket says 172.21.15.92:33096) 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 -- 172.21.15.92:0/773103749 learned_addr learned my addr 172.21.15.92:0/773103749 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69a1400640 1 --2- 172.21.15.92:0/773103749 >> v2:172.21.15.159:3300/0 conn(0x7f69ac0c6470 0x7f69ac0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69a9e00640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 -- 172.21.15.92:0/773103749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 msgr2=0x7f69ac06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:34.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac06eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 -- 172.21.15.92:0/773103749 >> v2:172.21.15.159:3300/0 conn(0x7f69ac0c6470 msgr2=0x7f69ac0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:34.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 --2- 172.21.15.92:0/773103749 >> v2:172.21.15.159:3300/0 conn(0x7f69ac0c6470 0x7f69ac0c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.076+0000 7f69aa800640 1 -- 172.21.15.92:0/773103749 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac0c9a80 con 0x7f69ac1593d0 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.077+0000 7f69aa800640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac1593d0 0x7f69ac0c44c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f69ac075310 tx=0x7f69a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199c7e5cfbe679ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.077+0000 7f69a9400640 1 -- 172.21.15.92:0/773103749 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a402c9e0 con 0x7f69ac1593d0 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.077+0000 7f69a9400640 1 -- 172.21.15.92:0/773103749 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a402cb80 con 0x7f69ac1593d0 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.077+0000 7f69a9400640 1 -- 172.21.15.92:0/773103749 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a40343f0 con 0x7f69ac1593d0 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 -- 172.21.15.92:0/773103749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac1593d0 msgr2=0x7f69ac0c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac1593d0 0x7f69ac0c44c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f69ac075310 tx=0x7f69a402be40 comp rx=0 tx=0).stop 2024-03-19T03:05:34.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 -- 172.21.15.92:0/773103749 shutdown_connections 2024-03-19T03:05:34.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac1593d0 0x7f69ac0c44c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 --2- 172.21.15.92:0/773103749 >> v2:172.21.15.159:3300/0 conn(0x7f69ac0c6470 0x7f69ac0c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 --2- 172.21.15.92:0/773103749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac06eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 -- 172.21.15.92:0/773103749 >> 172.21.15.92:0/773103749 conn(0x7f69ac0c8d90 msgr2=0x7f69ac0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:34.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 -- 172.21.15.92:0/773103749 shutdown_connections 2024-03-19T03:05:34.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.078+0000 7f69b0a00640 1 -- 172.21.15.92:0/773103749 wait complete. 2024-03-19T03:05:34.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.080+0000 7f69b0a00640 1 Processor -- start 2024-03-19T03:05:34.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.080+0000 7f69b0a00640 1 -- start start 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac071190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 0x7f69ac0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac0c8930 con 0x7f69ac0c6470 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac06ea60 con 0x7f69ac1593d0 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69b0a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f69ac06e6f0 con 0x7f69ac152bf0 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69a9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69a9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33100/0 (socket says 172.21.15.92:33100) 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69a9e00640 1 -- 172.21.15.92:0/1663996145 learned_addr learned my addr 172.21.15.92:0/1663996145 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69a1400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.081+0000 7f69aa800640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 0x7f69ac0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 -- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 msgr2=0x7f69ac071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 -- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 msgr2=0x7f69ac0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:34.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 0x7f69ac0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 -- 172.21.15.92:0/1663996145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac1f4ef0 con 0x7f69ac0c6470 2024-03-19T03:05:34.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69aa800640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 0x7f69ac0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:34.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a9e00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f699c00e9b0 tx=0x7f699c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c017070 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c00c9a0 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ac150c20 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.082+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c00cd90 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.083+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ac1511e0 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.084+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ac0684c0 con 0x7f69ac0c6470 2024-03-19T03:05:34.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.085+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f699c01a070 con 0x7f69ac0c6470 2024-03-19T03:05:34.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.085+0000 7f69a2a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 0x7f698806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:34.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.086+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f699c08ad50 con 0x7f69ac0c6470 2024-03-19T03:05:34.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.086+0000 7f69a1400640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 0x7f698806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:34.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.087+0000 7f69a1400640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 0x7f698806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f69ac14ce70 tx=0x7f6994009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:34.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.091+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f699c058e80 con 0x7f69ac0c6470 2024-03-19T03:05:34.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.326+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69ac06f950 con 0x7f69ac0c6470 2024-03-19T03:05:34.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.327+0000 7f69a2a00640 1 -- 172.21.15.92:0/1663996145 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6010 (secure 0 0 0) 0x7f699c05cf20 con 0x7f69ac0c6470 2024-03-19T03:05:34.329 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:34.329 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:32.135066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:34.330 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 35 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 msgr2=0x7f698806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 0x7f698806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f69ac14ce70 tx=0x7f6994009190 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 msgr2=0x7f69ac1f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f699c00e9b0 tx=0x7f699c00ee70 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 shutdown_connections 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f698806bbd0 0x7f698806e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69ac1593d0 0x7f69ac0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69ac152bf0 0x7f69ac071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 --2- 172.21.15.92:0/1663996145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69ac0c6470 0x7f69ac1f45c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:34.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 >> 172.21.15.92:0/1663996145 conn(0x7f69ac0c8d90 msgr2=0x7f69ac1565a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:34.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 shutdown_connections 2024-03-19T03:05:34.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:34.330+0000 7f69b0a00640 1 -- 172.21.15.92:0/1663996145 wait complete. 2024-03-19T03:05:34.350 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:36.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:36.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.568+0000 7f2481a00640 1 Processor -- start 2024-03-19T03:05:36.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.568+0000 7f2481a00640 1 -- start start 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2481a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c0733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2481a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f247c0c9010 0x7f247c0c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2481a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247c06f110 con 0x7f247c0c2b10 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2481a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f247c0c94c0 con 0x7f247c0c9010 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2481a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f247c071e50 con 0x7f247c151c00 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f247a000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f247c0c2b10 0x7f247c0b2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40470/0 (socket says 172.21.15.92:40470) 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f247a000640 1 -- 172.21.15.92:0/231207408 learned_addr learned my addr 172.21.15.92:0/231207408 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f247aa00640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c0733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f247b400640 1 --2- 172.21.15.92:0/231207408 >> v2:172.21.15.92:3300/0 conn(0x7f247c0c9010 0x7f247c0c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.570+0000 7f2479600640 1 -- 172.21.15.92:0/231207408 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3627395893 0 0) 0x7f247c06f110 con 0x7f247c0c2b10 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.571+0000 7f2479600640 1 -- 172.21.15.92:0/231207408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2460003520 con 0x7f247c0c2b10 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.571+0000 7f247b400640 1 -- 172.21.15.92:0/231207408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 msgr2=0x7f247c0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.571+0000 7f247b400640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c0733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.571+0000 7f247b400640 1 -- 172.21.15.92:0/231207408 >> v1:172.21.15.92:6789/0 conn(0x7f247c0c2b10 legacy=0x7f247c0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.571+0000 7f247b400640 1 -- 172.21.15.92:0/231207408 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c071890 con 0x7f247c0c9010 2024-03-19T03:05:36.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.572+0000 7f247b400640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c9010 0x7f247c0c2ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2474009d90 tx=0x7f247402e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79b197550def3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:36.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.572+0000 7f2479600640 1 -- 172.21.15.92:0/231207408 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247403b070 con 0x7f247c0c9010 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.572+0000 7f2479600640 1 -- 172.21.15.92:0/231207408 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2474036440 con 0x7f247c0c9010 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.572+0000 7f2479600640 1 -- 172.21.15.92:0/231207408 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247402ee60 con 0x7f247c0c9010 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 -- 172.21.15.92:0/231207408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c9010 msgr2=0x7f247c0c2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c9010 0x7f247c0c2ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2474009d90 tx=0x7f247402e0b0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 -- 172.21.15.92:0/231207408 shutdown_connections 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c9010 0x7f247c0c2ff0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 --2- 172.21.15.92:0/231207408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c0733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 -- 172.21.15.92:0/231207408 >> 172.21.15.92:0/231207408 conn(0x7f247c0ccb50 msgr2=0x7f247c0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:36.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 -- 172.21.15.92:0/231207408 shutdown_connections 2024-03-19T03:05:36.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.573+0000 7f2481a00640 1 -- 172.21.15.92:0/231207408 wait complete. 2024-03-19T03:05:36.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.574+0000 7f2481a00640 1 Processor -- start 2024-03-19T03:05:36.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.574+0000 7f2481a00640 1 -- start start 2024-03-19T03:05:36.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f247c0c9010 0x7f247c1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f247c06f690 con 0x7f247c0c2b10 2024-03-19T03:05:36.581 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f247c06f3b0 con 0x7f247c0c9010 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.575+0000 7f2481a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f247c151fe0 con 0x7f247c151c00 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45346/0 (socket says 172.21.15.92:45346) 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 -- 172.21.15.92:0/1788391854 learned_addr learned my addr 172.21.15.92:0/1788391854 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247b400640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247a000640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f247c0c9010 0x7f247c1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 -- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 msgr2=0x7f247c144c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c144c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 -- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f247c0c9010 msgr2=0x7f247c1446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f247c0c9010 0x7f247c1446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 -- 172.21.15.92:0/1788391854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c1e0de0 con 0x7f247c0c2b10 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247b400640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c144c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.576+0000 7f247aa00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e9b0 tx=0x7f246c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.577+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c017070 con 0x7f247c0c2b10 2024-03-19T03:05:36.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.577+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c00c9a0 con 0x7f247c0c2b10 2024-03-19T03:05:36.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.577+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c00cd90 con 0x7f247c0c2b10 2024-03-19T03:05:36.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.577+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c1e10c0 con 0x7f247c0c2b10 2024-03-19T03:05:36.584 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.577+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c15bce0 con 0x7f247c0c2b10 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.578+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f247c0768a0 con 0x7f247c0c2b10 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.579+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f246c01a070 con 0x7f247c0c2b10 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.580+0000 7f2472a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 0x7f245c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.580+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f246c08aff0 con 0x7f247c0c2b10 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.580+0000 7f247a000640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 0x7f245c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:36.586 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.581+0000 7f247a000640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 0x7f245c06e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f247c1ed380 tx=0x7f2468009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:36.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.585+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f246c0591f0 con 0x7f247c0c2b10 2024-03-19T03:05:36.824 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.823+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f247c0c8df0 con 0x7f247c0c2b10 2024-03-19T03:05:36.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.824+0000 7f2472a00640 1 -- 172.21.15.92:0/1788391854 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6010 (secure 0 0 0) 0x7f246c05f030 con 0x7f247c0c2b10 2024-03-19T03:05:36.825 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:36.825 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:32.135066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:36.826 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 35 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 msgr2=0x7f245c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 0x7f245c06e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f247c1ed380 tx=0x7f2468009040 comp rx=0 tx=0).stop 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 msgr2=0x7f247c1e2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e9b0 tx=0x7f246c00ee70 comp rx=0 tx=0).stop 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 shutdown_connections 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f245c06bbd0 0x7f245c06e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f247c151c00 0x7f247c144c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.828 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f247c0c9010 0x7f247c1446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 --2- 172.21.15.92:0/1788391854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f247c0c2b10 0x7f247c1e2ba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:36.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 >> 172.21.15.92:0/1788391854 conn(0x7f247c0ccb50 msgr2=0x7f247c0c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:36.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 shutdown_connections 2024-03-19T03:05:36.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:36.827+0000 7f2481a00640 1 -- 172.21.15.92:0/1788391854 wait complete. 2024-03-19T03:05:36.847 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T02:54:55.655938+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5347": { "addr": "172.21.15.159:6837/3372688839", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6835", "nonce": 3372688839, "type": "v2" }, { "addr": "172.21.15.159:6837", "nonce": 3372688839, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5347, "incarnation": 32, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:clientreplay", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T03:05:32.135066+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5347 } } } ], "standbys": [ { "addr": "172.21.15.92:6835/4145321899", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 4145321899, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 4145321899, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4188, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6837/2921443021", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6836", "nonce": 2921443021, "type": "v2" }, { "addr": "172.21.15.92:6837", "nonce": 2921443021, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.159:6836/3633923980", "addrs": { "addrvec": [ { "addr": "172.21.15.159:6834", "nonce": 3633923980, "type": "v2" }, { "addr": "172.21.15.159:6836", "nonce": 3633923980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.92:6833/3249321834", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6832", "nonce": 3249321834, "type": "v2" }, { "addr": "172.21.15.92:6833", "nonce": 3249321834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-19T03:05:38.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:39.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.076+0000 7ff167200640 1 Processor -- start 2024-03-19T03:05:39.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.076+0000 7ff167200640 1 -- start start 2024-03-19T03:05:39.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.077+0000 7ff167200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 0x7ff168152c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:39.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.077+0000 7ff167200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1680c6d00 con 0x7ff1680c5f50 2024-03-19T03:05:39.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.077+0000 7ff167200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16806d610 con 0x7ff1680c5b70 2024-03-19T03:05:39.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.077+0000 7ff167200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff16815c580 con 0x7ff16806d160 2024-03-19T03:05:39.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff165e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff1680c5b70 0x7ff16815be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:42030/0 (socket says 172.21.15.92:42030) 2024-03-19T03:05:39.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff165400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff1680c5f50 0x7ff168152520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40476/0 (socket says 172.21.15.92:40476) 2024-03-19T03:05:39.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff165400640 1 -- 172.21.15.92:0/3398888469 learned_addr learned my addr 172.21.15.92:0/3398888469 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:39.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff166800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 0x7ff168152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161801741 0 0) 0x7ff1680c6d00 con 0x7ff1680c5f50 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff14c003520 con 0x7ff1680c5f50 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244467972 0 0) 0x7ff16806d610 con 0x7ff1680c5b70 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1680c6d00 con 0x7ff1680c5b70 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff166800640 1 -- 172.21.15.92:0/3398888469 >> v1:172.21.15.159:6789/0 conn(0x7ff1680c5b70 legacy=0x7ff16815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff166800640 1 -- 172.21.15.92:0/3398888469 >> v1:172.21.15.92:6789/0 conn(0x7ff1680c5f50 legacy=0x7ff168152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.078+0000 7ff166800640 1 -- 172.21.15.92:0/3398888469 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1680c6fa0 con 0x7ff16806d160 2024-03-19T03:05:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.079+0000 7ff166800640 1 --2- 172.21.15.92:0/3398888469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 0x7ff168152c30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff160009dc0 tx=0x7ff16002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fdc897b101a5289 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160039070 con 0x7ff16806d160 2024-03-19T03:05:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff16002cd50 con 0x7ff16806d160 2024-03-19T03:05:39.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff164a00640 1 -- 172.21.15.92:0/3398888469 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160034660 con 0x7ff16806d160 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 -- 172.21.15.92:0/3398888469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 msgr2=0x7ff168152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 --2- 172.21.15.92:0/3398888469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 0x7ff168152c30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff160009dc0 tx=0x7ff16002bfa0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 -- 172.21.15.92:0/3398888469 shutdown_connections 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 --2- 172.21.15.92:0/3398888469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff16806d160 0x7ff168152c30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 -- 172.21.15.92:0/3398888469 >> 172.21.15.92:0/3398888469 conn(0x7ff1680cc1f0 msgr2=0x7ff1680cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 -- 172.21.15.92:0/3398888469 shutdown_connections 2024-03-19T03:05:39.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.080+0000 7ff167200640 1 -- 172.21.15.92:0/3398888469 wait complete. 2024-03-19T03:05:39.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.082+0000 7ff167200640 1 Processor -- start 2024-03-19T03:05:39.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.082+0000 7ff167200640 1 -- start start 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 0x7ff1681590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1680c5b70 0x7ff16814d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff16806dbb0 con 0x7ff16806d160 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff16806df20 con 0x7ff1680c5b70 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff167200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff1680729b0 con 0x7ff1680c5f50 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff165e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 0x7ff1681590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:39.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff166800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:39.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff166800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:32992/0 (socket says 172.21.15.92:32992) 2024-03-19T03:05:39.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff166800640 1 -- 172.21.15.92:0/981211574 learned_addr learned my addr 172.21.15.92:0/981211574 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:39.085 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.083+0000 7ff165400640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1680c5b70 0x7ff16814d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 -- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1680c5b70 msgr2=0x7ff16814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1680c5b70 0x7ff16814d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 -- 172.21.15.92:0/981211574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 msgr2=0x7ff1681590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 0x7ff1681590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 -- 172.21.15.92:0/981211574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff168157f70 con 0x7ff1680c5f50 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff165e00640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 0x7ff1681590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.084+0000 7ff166800640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff160037040 tx=0x7ff160004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.085+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160039050 con 0x7ff1680c5f50 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.085+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff160045070 con 0x7ff1680c5f50 2024-03-19T03:05:39.086 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.085+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160009450 con 0x7ff1680c5f50 2024-03-19T03:05:39.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.085+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff168158250 con 0x7ff1680c5f50 2024-03-19T03:05:39.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.085+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff168158810 con 0x7ff1680c5f50 2024-03-19T03:05:39.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.086+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff168158a10 con 0x7ff1680c5f50 2024-03-19T03:05:39.087 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.087+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff16003e850 con 0x7ff1680c5f50 2024-03-19T03:05:39.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.088+0000 7ff15e000640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 0x7ff14806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:39.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.088+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7ff1600addc0 con 0x7ff1680c5f50 2024-03-19T03:05:39.089 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.088+0000 7ff165e00640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 0x7ff14806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:39.089 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.089+0000 7ff165e00640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 0x7ff14806e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff1580011e0 tx=0x7ff158007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:39.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.094+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff168158a10 con 0x7ff1680c5f50 2024-03-19T03:05:39.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.340+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff16804f580 con 0x7ff1680c5f50 2024-03-19T03:05:39.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.341+0000 7ff15e000640 1 -- 172.21.15.92:0/981211574 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6010 (secure 0 0 0) 0x7ff16007c1b0 con 0x7ff1680c5f50 2024-03-19T03:05:39.342 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:39.342 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:32.135066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-19T03:05:39.342 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 35 2024-03-19T03:05:39.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.343+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 msgr2=0x7ff14806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.343+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 0x7ff14806e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff1580011e0 tx=0x7ff158007d30 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 msgr2=0x7ff16814df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff160037040 tx=0x7ff160004a40 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 shutdown_connections 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff14806bbd0 0x7ff14806e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff1680c5f50 0x7ff16814df20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1680c5b70 0x7ff16814d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 --2- 172.21.15.92:0/981211574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16806d160 0x7ff1681590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 >> 172.21.15.92:0/981211574 conn(0x7ff1680cc1f0 msgr2=0x7ff1680cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:39.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 shutdown_connections 2024-03-19T03:05:39.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:39.344+0000 7ff167200640 1 -- 172.21.15.92:0/981211574 wait complete. 2024-03-19T03:05:39.365 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-19T03:05:41.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:41.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.584+0000 7fd307c00640 1 Processor -- start 2024-03-19T03:05:41.585 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.585+0000 7fd307c00640 1 -- start start 2024-03-19T03:05:41.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd307c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd300071190 0x7fd30006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd307c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd30006f270 0x7fd300161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd307c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd300071d90 con 0x7fd300151260 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd307c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd3000c6c90 con 0x7fd300071190 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd307c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd300071620 con 0x7fd30006f270 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.586+0000 7fd2f7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd300071190 0x7fd30006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd300151260 0x7fd3000c76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:42044/0 (socket says 172.21.15.92:42044) 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305200640 1 -- 172.21.15.92:0/3820248897 learned_addr learned my addr 172.21.15.92:0/3820248897 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305c00640 1 --2- 172.21.15.92:0/3820248897 >> v2:172.21.15.159:3300/0 conn(0x7fd30006f270 0x7fd300161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd2ffe00640 1 -- 172.21.15.92:0/3820248897 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865370502 0 0) 0x7fd300071d90 con 0x7fd300151260 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd2ffe00640 1 -- 172.21.15.92:0/3820248897 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2e4003520 con 0x7fd300151260 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305c00640 1 -- 172.21.15.92:0/3820248897 >> v2:172.21.15.92:3300/0 conn(0x7fd300071190 msgr2=0x7fd30006ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:41.588 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305c00640 1 --2- 172.21.15.92:0/3820248897 >> v2:172.21.15.92:3300/0 conn(0x7fd300071190 0x7fd30006ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305c00640 1 -- 172.21.15.92:0/3820248897 >> v1:172.21.15.159:6789/0 conn(0x7fd300151260 legacy=0x7fd3000c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.587+0000 7fd305c00640 1 -- 172.21.15.92:0/3820248897 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3000c7df0 con 0x7fd30006f270 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.588+0000 7fd305c00640 1 --2- 172.21.15.92:0/3820248897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30006f270 0x7fd300161840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd300071120 tx=0x7fd2f802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d35cc38b90ffc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.588+0000 7fd2ffe00640 1 -- 172.21.15.92:0/3820248897 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8039070 con 0x7fd30006f270 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.588+0000 7fd2ffe00640 1 -- 172.21.15.92:0/3820248897 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f802cd00 con 0x7fd30006f270 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.588+0000 7fd2ffe00640 1 -- 172.21.15.92:0/3820248897 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8034660 con 0x7fd30006f270 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 -- 172.21.15.92:0/3820248897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30006f270 msgr2=0x7fd300161840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 --2- 172.21.15.92:0/3820248897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30006f270 0x7fd300161840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd300071120 tx=0x7fd2f802bf50 comp rx=0 tx=0).stop 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 -- 172.21.15.92:0/3820248897 shutdown_connections 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 --2- 172.21.15.92:0/3820248897 >> v2:172.21.15.92:3300/0 conn(0x7fd300071190 0x7fd30006ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 --2- 172.21.15.92:0/3820248897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd30006f270 0x7fd300161840 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 -- 172.21.15.92:0/3820248897 >> 172.21.15.92:0/3820248897 conn(0x7fd3000c8d90 msgr2=0x7fd3000c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:41.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 -- 172.21.15.92:0/3820248897 shutdown_connections 2024-03-19T03:05:41.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.589+0000 7fd307c00640 1 -- 172.21.15.92:0/3820248897 wait complete. 2024-03-19T03:05:41.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.591+0000 7fd307c00640 1 Processor -- start 2024-03-19T03:05:41.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.591+0000 7fd307c00640 1 -- start start 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 0x7fd3001f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 0x7fd3001e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd3000c6ee0 con 0x7fd300151260 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd300157d20 con 0x7fd300071190 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd307c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd3000c71c0 con 0x7fd30006f270 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd305c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd305200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 0x7fd3001f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 0x7fd3001e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd305c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45374/0 (socket says 172.21.15.92:45374) 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd305c00640 1 -- 172.21.15.92:0/3421414221 learned_addr learned my addr 172.21.15.92:0/3421414221 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.592+0000 7fd305200640 1 -- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 msgr2=0x7fd3001e0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305200640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 0x7fd3001e0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305200640 1 -- 172.21.15.92:0/3421414221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 msgr2=0x7fd30015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305200640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305200640 1 -- 172.21.15.92:0/3421414221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3001f0c40 con 0x7fd30006f270 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd2f7e00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 0x7fd3001e0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd305200640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 0x7fd3001f0540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c430 tx=0x7fd2f000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0017070 con 0x7fd30006f270 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f0005ce0 con 0x7fd30006f270 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3001f0f00 con 0x7fd30006f270 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00072f0 con 0x7fd30006f270 2024-03-19T03:05:41.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.593+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd300159dd0 con 0x7fd30006f270 2024-03-19T03:05:41.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.594+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3000684c0 con 0x7fd30006f270 2024-03-19T03:05:41.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.595+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd2f001a070 con 0x7fd30006f270 2024-03-19T03:05:41.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.596+0000 7fd2fd600640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 0x7fd2dc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:41.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.596+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fd2f0089f40 con 0x7fd30006f270 2024-03-19T03:05:41.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.596+0000 7fd2f7e00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 0x7fd2dc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:41.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.597+0000 7fd2f7e00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 0x7fd2dc06e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b240 tx=0x7fd2e800b3a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:41.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.600+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd2f0007e20 con 0x7fd30006f270 2024-03-19T03:05:41.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.838+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3000c83d0 con 0x7fd30006f270 2024-03-19T03:05:41.840 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.839+0000 7fd2fd600640 1 -- 172.21.15.92:0/3421414221 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6011 (secure 0 0 0) 0x7fd2f0058190 con 0x7fd30006f270 2024-03-19T03:05:41.840 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:41.840 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:41.531709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5347,"qdb_cluster":[5347]},"id":1}]} 2024-03-19T03:05:41.840 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 36 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 msgr2=0x7fd2dc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 0x7fd2dc06e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b240 tx=0x7fd2e800b3a0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 msgr2=0x7fd3001f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 0x7fd3001f0540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c430 tx=0x7fd2f000c8f0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 shutdown_connections 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd2dc06bc00 0x7fd2dc06e0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd300151260 0x7fd30015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd300071190 0x7fd3001e0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 --2- 172.21.15.92:0/3421414221 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd30006f270 0x7fd3001f0540 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 >> 172.21.15.92:0/3421414221 conn(0x7fd3000c8d90 msgr2=0x7fd3001e1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:41.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 shutdown_connections 2024-03-19T03:05:41.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:41.842+0000 7fd307c00640 1 -- 172.21.15.92:0/3421414221 wait complete. 2024-03-19T03:05:41.863 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=5185 2024-03-19T03:05:41.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2024-03-19T03:05:48.209 INFO:tasks.ceph.mds.f.smithi159.stderr:2024-03-19T03:05:48.207+0000 7f22665a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T03:05:48.864 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-03-19T03:05:48.864 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-19T03:05:48.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-19T03:05:48.867 INFO:tasks.ceph.mds.d:Started 2024-03-19T03:05:48.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:48.941 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-03-19T03:05:49.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.089+0000 7f7697200640 1 Processor -- start 2024-03-19T03:05:49.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.089+0000 7f7697200640 1 -- start start 2024-03-19T03:05:49.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.090+0000 7f7697200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.090+0000 7f7697200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7690072890 con 0x7f769006f800 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.090+0000 7f7697200640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76900c8200 con 0x7f7690158db0 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.090+0000 7f7697200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76900c6c90 con 0x7f7690071da0 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:35032/0 (socket says 172.21.15.92:35032) 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 -- 172.21.15.92:0/2226714518 learned_addr learned my addr 172.21.15.92:0/2226714518 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 -- 172.21.15.92:0/2226714518 >> v1:172.21.15.159:6789/0 conn(0x7f7690158db0 legacy=0x7f76900c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 -- 172.21.15.92:0/2226714518 >> v1:172.21.15.92:6789/0 conn(0x7f769006f800 legacy=0x7f7690072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768fe00640 1 -- 172.21.15.92:0/2226714518 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76900c84e0 con 0x7f7690071da0 2024-03-19T03:05:49.093 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.091+0000 7f768f400640 1 -- 172.21.15.92:0/2226714518 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685154371 0 0) 0x7f76900c8200 con 0x7f7690158db0 2024-03-19T03:05:49.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.092+0000 7f768fe00640 1 --2- 172.21.15.92:0/2226714518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76800098a0 tx=0x7f768002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26f88b9c734a1c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:49.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.092+0000 7f768f400640 1 -- 172.21.15.92:0/2226714518 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768002f830 con 0x7f7690071da0 2024-03-19T03:05:49.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.092+0000 7f768f400640 1 -- 172.21.15.92:0/2226714518 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768002f9d0 con 0x7f7690071da0 2024-03-19T03:05:49.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f768f400640 1 -- 172.21.15.92:0/2226714518 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768002fda0 con 0x7f7690071da0 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 -- 172.21.15.92:0/2226714518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 msgr2=0x7f76900c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 --2- 172.21.15.92:0/2226714518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76800098a0 tx=0x7f768002ba10 comp rx=0 tx=0).stop 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 -- 172.21.15.92:0/2226714518 shutdown_connections 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 --2- 172.21.15.92:0/2226714518 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7690071da0 0x7f76900c8740 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 -- 172.21.15.92:0/2226714518 >> 172.21.15.92:0/2226714518 conn(0x7f76900c8d90 msgr2=0x7f76900c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 -- 172.21.15.92:0/2226714518 shutdown_connections 2024-03-19T03:05:49.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.093+0000 7f7697200640 1 -- 172.21.15.92:0/2226714518 wait complete. 2024-03-19T03:05:49.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.095+0000 7f7697200640 1 Processor -- start 2024-03-19T03:05:49.096 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.095+0000 7f7697200640 1 -- start start 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 0x7f76901f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f76900c7830 con 0x7f7690071da0 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f769015fde0 con 0x7f7690158db0 2024-03-19T03:05:49.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7697200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76900c79b0 con 0x7f769006f800 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7687400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f768fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7687400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51358/0 (socket says 172.21.15.92:51358) 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f768fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:35040/0 (socket says 172.21.15.92:35040) 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7687400640 1 -- 172.21.15.92:0/496656297 learned_addr learned my addr 172.21.15.92:0/496656297 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.096+0000 7f7695200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 0x7f76901f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 -- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 msgr2=0x7f76901f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 0x7f76901f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 -- 172.21.15.92:0/496656297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 msgr2=0x7f769015e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 -- 172.21.15.92:0/496656297 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76901f4b20 con 0x7f769006f800 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f7687400640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:49.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f7695200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 0x7f76901f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768fe00640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7680033000 tx=0x7f7680004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680037d40 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7680041400 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76800417f0 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769015e120 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.097+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76901f4de0 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.098+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7690071920 con 0x7f769006f800 2024-03-19T03:05:49.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.100+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7680048030 con 0x7f769006f800 2024-03-19T03:05:49.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.100+0000 7f768cc00640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 0x7f766c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:49.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.101+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f76800ad600 con 0x7f769006f800 2024-03-19T03:05:49.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.101+0000 7f7687400640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 0x7f766c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:49.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.101+0000 7f7687400640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 0x7f766c06e0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f769015a880 tx=0x7f7678009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:49.106 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.105+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7690071920 con 0x7f769006f800 2024-03-19T03:05:49.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.339+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76900684c0 con 0x7f769006f800 2024-03-19T03:05:49.341 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.340+0000 7f768cc00640 1 -- 172.21.15.92:0/496656297 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6011 (secure 0 0 0) 0x7f768004f060 con 0x7f769006f800 2024-03-19T03:05:49.342 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:49.342 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:41.531709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5347,"qdb_cluster":[5347]},"id":1}]} 2024-03-19T03:05:49.342 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 36 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.342+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 msgr2=0x7f766c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 0x7f766c06e0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f769015a880 tx=0x7f7678009040 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 msgr2=0x7f769015cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7680033000 tx=0x7f7680004730 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 shutdown_connections 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f766c06bc00 0x7f766c06e0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7690158db0 0x7f76901f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7690071da0 0x7f769015e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 --2- 172.21.15.92:0/496656297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f769006f800 0x7f769015cde0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 >> 172.21.15.92:0/496656297 conn(0x7f76900c8d90 msgr2=0x7f7690157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 shutdown_connections 2024-03-19T03:05:49.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:49.343+0000 7f7697200640 1 -- 172.21.15.92:0/496656297 wait complete. 2024-03-19T03:05:49.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-19T03:05:51.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:05:51.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.585+0000 7f5aa6200640 1 Processor -- start 2024-03-19T03:05:51.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.585+0000 7f5aa6200640 1 -- start start 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.586+0000 7f5aa6200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5aa0154ec0 0x7f5aa015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5aa6200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5aa00cdcc0 0x7f5aa006cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5aa6200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa006d100 0x7f5aa00cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5aa6200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa015f000 con 0x7f5aa00cdcc0 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5aa6200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa01552a0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5aa6200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa015e480 con 0x7f5aa006d100 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5aa0154ec0 0x7f5aa015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa006d100 0x7f5aa00cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5aa0154ec0 0x7f5aa015ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:59924/0 (socket says 172.21.15.92:59924) 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 -- 172.21.15.92:0/512105704 learned_addr learned my addr 172.21.15.92:0/512105704 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9ea00640 1 --2- 172.21.15.92:0/512105704 >> v2:172.21.15.92:3300/0 conn(0x7f5aa00cdcc0 0x7f5aa006cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 -- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa006d100 msgr2=0x7f5aa00cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 --2- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa006d100 0x7f5aa00cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 -- 172.21.15.92:0/512105704 >> v2:172.21.15.92:3300/0 conn(0x7f5aa00cdcc0 msgr2=0x7f5aa006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 --2- 172.21.15.92:0/512105704 >> v2:172.21.15.92:3300/0 conn(0x7f5aa00cdcc0 0x7f5aa006cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.587+0000 7f5a9f400640 1 -- 172.21.15.92:0/512105704 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa015e760 con 0x7f5aa0154ec0 2024-03-19T03:05:51.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.588+0000 7f5a9f400640 1 --2- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa015ab50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00cd7a0 tx=0x7f5a9002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99aeae10cf9538b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:51.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5a9e000640 1 -- 172.21.15.92:0/512105704 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90035020 con 0x7f5aa0154ec0 2024-03-19T03:05:51.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5a9e000640 1 -- 172.21.15.92:0/512105704 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a90031870 con 0x7f5aa0154ec0 2024-03-19T03:05:51.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5a9e000640 1 -- 172.21.15.92:0/512105704 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90031c40 con 0x7f5aa0154ec0 2024-03-19T03:05:51.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 -- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 msgr2=0x7f5aa015ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 --2- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa015ab50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00cd7a0 tx=0x7f5a9002b910 comp rx=0 tx=0).stop 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 -- 172.21.15.92:0/512105704 shutdown_connections 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 --2- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa006d100 0x7f5aa00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 --2- 172.21.15.92:0/512105704 >> v2:172.21.15.92:3300/0 conn(0x7f5aa00cdcc0 0x7f5aa006cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 --2- 172.21.15.92:0/512105704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa015ab50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 -- 172.21.15.92:0/512105704 >> 172.21.15.92:0/512105704 conn(0x7f5aa006da80 msgr2=0x7f5aa006c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.589+0000 7f5aa6200640 1 -- 172.21.15.92:0/512105704 shutdown_connections 2024-03-19T03:05:51.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.590+0000 7f5aa6200640 1 -- 172.21.15.92:0/512105704 wait complete. 2024-03-19T03:05:51.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.591+0000 7f5aa6200640 1 Processor -- start 2024-03-19T03:05:51.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.591+0000 7f5aa6200640 1 -- start start 2024-03-19T03:05:51.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5aa006d100 0x7f5aa015db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 0x7f5aa00671c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa00cbd10 con 0x7f5aa006d100 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa015ed20 con 0x7f5aa0154ec0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5aa6200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa00c7550 con 0x7f5aa00cdcc0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 0x7f5aa00671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5aa006d100 0x7f5aa015db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:59934/0 (socket says 172.21.15.92:59934) 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9fe00640 1 -- 172.21.15.92:0/4203964825 learned_addr learned my addr 172.21.15.92:0/4203964825 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.592+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 0x7f5aa00671c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:35066/0 (socket says 172.21.15.92:35066) 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 -- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 msgr2=0x7f5aa00671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 0x7f5aa00671c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 -- 172.21.15.92:0/4203964825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5aa006d100 msgr2=0x7f5aa015db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5aa006d100 0x7f5aa015db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 -- 172.21.15.92:0/4203964825 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa01f0de0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a9fe00640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800eed0 tx=0x7f5a9800b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:51.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98018020 con 0x7f5aa0154ec0 2024-03-19T03:05:51.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a98014400 con 0x7f5aa0154ec0 2024-03-19T03:05:51.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa01f10a0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a980147f0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.593+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa01f1660 con 0x7f5aa0154ec0 2024-03-19T03:05:51.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.594+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa00684c0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.595+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5a98005d10 con 0x7f5aa0154ec0 2024-03-19T03:05:51.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.596+0000 7f5a97400640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 0x7f5a8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:05:51.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.596+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f5a9808b5c0 con 0x7f5aa0154ec0 2024-03-19T03:05:51.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.596+0000 7f5a9f400640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 0x7f5a8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:05:51.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.597+0000 7f5a9f400640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 0x7f5a8006e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a90036000 tx=0x7f5a9003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:05:51.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.601+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5a98022030 con 0x7f5aa0154ec0 2024-03-19T03:05:51.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.902+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5aa00c6680 con 0x7f5aa0154ec0 2024-03-19T03:05:51.904 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.903+0000 7f5a97400640 1 -- 172.21.15.92:0/4203964825 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6839 (secure 0 0 0) 0x7f5a98059780 con 0x7f5aa0154ec0 2024-03-19T03:05:51.905 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:05:51.905 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5506,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3348696376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3348696376},{"type":"v1","addr":"172.21.15.159:6839","nonce":3348696376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:41.531709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5347,"qdb_cluster":[5347]},"id":1}]} 2024-03-19T03:05:51.905 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 37 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 msgr2=0x7f5a8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 0x7f5a8006e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a90036000 tx=0x7f5a9003c040 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 msgr2=0x7f5aa00c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800eed0 tx=0x7f5a9800b480 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 shutdown_connections 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5a8006bbd0 0x7f5a8006e090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5aa0154ec0 0x7f5aa00c7820 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5aa00cdcc0 0x7f5aa00671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 --2- 172.21.15.92:0/4203964825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5aa006d100 0x7f5aa015db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:05:51.909 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 >> 172.21.15.92:0/4203964825 conn(0x7f5aa006da80 msgr2=0x7f5aa01531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:05:51.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 shutdown_connections 2024-03-19T03:05:51.910 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:05:51.907+0000 7f5aa6200640 1 -- 172.21.15.92:0/4203964825 wait complete. 2024-03-19T03:05:51.927 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-03-19T03:05:51.928 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2024-03-19T03:06:02.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:02.200 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:05.463 INFO:tasks.workunit.client.0.smithi092.stderr:Updating files: 1% (165/12365) Updating files: 1% (208/12365) Updating files: 2% (248/12365) Updating files: 2% (358/12365) Updating files: 3% (371/12365) Updating files: 4% (495/12365) Updating files: 4% (521/12365) Updating files: 4% (601/12365) Updating files: 5% (619/12365) Updating files: 5% (628/12365) Updating files: 5% (729/12365) Updating files: 5% (732/12365) Updating files: 6% (742/12365) Updating files: 6% (820/12365) Updating files: 7% (866/12365) Updating files: 7% (884/12365) Updating files: 7% (895/12365) Updating files: 8% (990/12365) Updating files: 8% (991/12365) Updating files: 8% (1112/12365) Updating files: 9% (1113/12365) Updating files: 10% (1237/12365) Updating files: 10% (1302/12365) Updating files: 11% (1361/12365) Updating files: 11% (1447/12365) Updating files: 12% (1484/12365) Updating files: 13% (1608/12365) Updating files: 13% (1626/12365) Updating files: 14% (1732/12365) Updating files: 14% (1797/12365) Updating files: 15% (1855/12365) Updating files: 15% (1956/12365) Updating files: 16% (1979/12365) Updating files: 17% (2103/12365) Updating files: 18% (2226/12365) Updating files: 18% (2335/12365) Updating files: 19% (2350/12365) Updating files: 19% (2402/12365) Updating files: 20% (2473/12365) Updating files: 20% (2488/12365) Updating files: 20% (2517/12365) Updating files: 21% (2597/12365) Updating files: 22% (2721/12365) Updating files: 22% (2798/12365) Updating files: 23% (2844/12365) Updating files: 23% (2949/12365) Updating files: 24% (2968/12365) Updating files: 24% (3051/12365) Updating files: 25% (3092/12365) Updating files: 26% (3215/12365) Updating files: 26% (3249/12365) Updating files: 26% (3324/12365) Updating files: 27% (3339/12365) Updating files: 27% (3365/12365) Updating files: 28% (3463/12365) Updating files: 28% (3543/12365) Updating files: 29% (3586/12365) Updating files: 29% (3691/12365) Updating files: 29% (3692/12365) Updating files: 30% (3710/12365) Updating files: 30% (3792/12365) Updating files: 30% (3813/12365) Updating files: 31% (3834/12365) Updating files: 32% (3957/12365) Updating files: 32% (3997/12365) Updating files: 32% (4066/12365) Updating files: 33% (4081/12365) Updating files: 33% (4166/12365) Updating files: 34% (4205/12365) Updating files: 34% (4242/12365) Updating files: 35% (4328/12365) Updating files: 35% (4336/12365) Updating files: 36% (4452/12365) Updating files: 36% (4466/12365) Updating files: 37% (4576/12365) Updating files: 38% (4699/12365) Updating files: 38% (4808/12365) Updating files: 39% (4823/12365) Updating files: 40% (4946/12365) Updating files: 40% (4959/12365) Updating files: 40% (5067/12365) Updating files: 41% (5070/12365) Updating files: 41% (5173/12365) Updating files: 42% (5194/12365) Updating files: 42% (5276/12365) Updating files: 42% (5312/12365) Updating files: 43% (5317/12365) Updating files: 43% (5339/12365) Updating files: 44% (5441/12365) Updating files: 44% (5483/12365) Updating files: 45% (5565/12365) Updating files: 45% (5571/12365) Updating files: 46% (5688/12365) Updating files: 46% (5785/12365) Updating files: 46% (5797/12365) Updating files: 47% (5812/12365) Updating files: 47% (5901/12365) Updating files: 48% (5936/12365) Updating files: 48% (6031/12365) Updating files: 49% (6059/12365) Updating files: 49% (6154/12365) Updating files: 49% (6178/12365) Updating files: 50% (6183/12365) Updating files: 50% (6219/12365) Updating files: 50% (6261/12365) Updating files: 51% (6307/12365) Updating files: 51% (6356/12365) Updating files: 52% (6430/12365) Updating files: 53% (6554/12365) Updating files: 53% (6667/12365) Updating files: 54% (6678/12365) Updating files: 55% (6801/12365) Updating files: 56% (6925/12365) Updating files: 56% (7034/12365) Updating files: 57% (7049/12365) Updating files: 58% (7172/12365) Updating files: 58% (7284/12365) Updating files: 59% (7296/12365) Updating files: 60% (7419/12365) Updating files: 60% (7476/12365) Updating files: 60% (7537/12365) Updating files: 60% (7542/12365) Updating files: 61% (7543/12365) Updating files: 62% (7667/12365) Updating files: 63% (7790/12365) Updating files: 63% (7801/12365) Updating files: 64% (7914/12365) Updating files: 64% (7951/12365) Updating files: 65% (8038/12365) Updating files: 65% (8096/12365) Updating files: 65% (8105/12365) Updating files: 65% (8106/12365) Updating files: 65% (8126/12365) Updating files: 66% (8161/12365) Updating files: 66% (8241/12365) Updating files: 67% (8285/12365) Updating files: 67% (8347/12365) Updating files: 68% (8409/12365) Updating files: 68% (8515/12365) Updating files: 69% (8532/12365) Updating files: 70% (8656/12365) Updating files: 70% (8761/12365) Updating files: 71% (8780/12365) Updating files: 71% (8850/12365) Updating files: 72% (8903/12365) Updating files: 72% (8944/12365) Updating files: 72% (8950/12365) Updating files: 72% (9023/12365) Updating files: 73% (9027/12365) Updating files: 73% (9145/12365) Updating files: 74% (9151/12365) Updating files: 74% (9271/12365) Updating files: 75% (9274/12365) Updating files: 75% (9286/12365) Updating files: 76% (9398/12365) Updating files: 76% (9518/12365) Updating files: 77% (9522/12365) Updating files: 77% (9575/12365) Updating files: 78% (9645/12365) Updating files: 79% (9769/12365) Updating files: 79% (9797/12365) Updating files: 79% (9860/12365) Updating files: 80% (9892/12365) Updating files: 80% (9953/12365) Updating files: 81% (10016/12365) Updating files: 81% (10118/12365) Updating files: 81% (10121/12365) Updating files: 81% (10122/12365) Updating files: 82% (10140/12365) Updating files: 82% (10149/12365) Updating files: 82% (10158/12365) Updating files: 82% (10244/12365) Updating files: 83% (10263/12365) Updating files: 84% (10387/12365) Updating files: 84% (10455/12365) Updating files: 84% (10491/12365) Updating files: 85% (10511/12365) Updating files: 85% (10627/12365) Updating files: 86% (10634/12365) Updating files: 86% (10728/12365) Updating files: 87% (10758/12365) Updating files: 88% (10882/12365) Updating files: 88% (10980/12365) Updating files: 89% (11005/12365) Updating files: 90% (11129/12365) Updating files: 91% (11253/12365) Updating files: 91% (11310/12365) Updating files: 92% (11376/12365) Updating files: 92% (11411/12365) Updating files: 93% (11500/12365) Updating files: 93% (11520/12365) Updating files: 94% (11624/12365) Updating files: 94% (11635/12365) Updating files: 94% (11712/12365) Updating files: 95% (11747/12365) Updating files: 96% (11871/12365) Updating files: 96% (11978/12365) Updating files: 97% (11995/12365) Updating files: 98% (12118/12365) Updating files: 98% (12142/12365) Updating files: 98% (12235/12365) Updating files: 99% (12242/12365) Updating files: 99% (12277/12365) Updating files: 99% (12354/12365) Updating files: 100% (12365/12365) Updating files: 100% (12365/12365), done. 2024-03-19T03:06:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-19T03:06:05.678 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-03-19T03:06:05.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-19T03:06:06.692 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi092 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-19T03:06:06.946 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=002a509fa8924b4d96d44ffdfcd8115e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi092 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-19T03:06:06.948 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-03-19T03:06:06.958 INFO:tasks.mds_thrash:joining mds_thrasher 2024-03-19T03:06:06.960 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-03-19T03:06:06.960 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-03-19T03:06:06.960 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 5 2024-03-19T03:06:06.961 INFO:tasks.mds_thrash:done joining 2024-03-19T03:06:06.961 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-19T03:06:07.002 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-19T03:06:07.002 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T03:06:07.003 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:06:07.062 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-19T03:06:07.063 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:06:07.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:06:07.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.326+0000 7fafb2600640 1 Processor -- start 2024-03-19T03:06:07.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.326+0000 7fafb2600640 1 -- start start 2024-03-19T03:06:07.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafb2600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 0x7fafac06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafb2600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafac0c9a80 con 0x7fafac0c6470 2024-03-19T03:06:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafb2600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafac0c8650 con 0x7fafac1593d0 2024-03-19T03:06:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafb2600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fafac0c6c90 con 0x7fafac152bf0 2024-03-19T03:06:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafaaa00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fafac0c6470 0x7fafac0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53788/0 (socket says 172.21.15.92:53788) 2024-03-19T03:06:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafab400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 0x7fafac06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafaaa00640 1 -- 172.21.15.92:0/1389703156 learned_addr learned my addr 172.21.15.92:0/1389703156 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:07.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafab400640 1 -- 172.21.15.92:0/1389703156 >> v1:172.21.15.159:6789/0 conn(0x7fafac1593d0 legacy=0x7fafac0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafab400640 1 -- 172.21.15.92:0/1389703156 >> v1:172.21.15.92:6789/0 conn(0x7fafac0c6470 legacy=0x7fafac0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafaa000640 1 -- 172.21.15.92:0/1389703156 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177995840 0 0) 0x7fafac0c9a80 con 0x7fafac0c6470 2024-03-19T03:06:07.331 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.328+0000 7fafab400640 1 -- 172.21.15.92:0/1389703156 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac0c8930 con 0x7fafac152bf0 2024-03-19T03:06:07.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.329+0000 7fafab400640 1 --2- 172.21.15.92:0/1389703156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 0x7fafac06eef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0097d0 tx=0x7faf9c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2375d7a58d978e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafaa000640 1 -- 172.21.15.92:0/1389703156 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c035020 con 0x7fafac152bf0 2024-03-19T03:06:07.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafaa000640 1 -- 172.21.15.92:0/1389703156 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9c031800 con 0x7fafac152bf0 2024-03-19T03:06:07.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafaa000640 1 -- 172.21.15.92:0/1389703156 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c031bd0 con 0x7fafac152bf0 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 -- 172.21.15.92:0/1389703156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 msgr2=0x7fafac06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 --2- 172.21.15.92:0/1389703156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 0x7fafac06eef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0097d0 tx=0x7faf9c02b8a0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 -- 172.21.15.92:0/1389703156 shutdown_connections 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 --2- 172.21.15.92:0/1389703156 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac152bf0 0x7fafac06eef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 -- 172.21.15.92:0/1389703156 >> 172.21.15.92:0/1389703156 conn(0x7fafac0c8d90 msgr2=0x7fafac0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 -- 172.21.15.92:0/1389703156 shutdown_connections 2024-03-19T03:06:07.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.330+0000 7fafb2600640 1 -- 172.21.15.92:0/1389703156 wait complete. 2024-03-19T03:06:07.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.332+0000 7fafb2600640 1 Processor -- start 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.332+0000 7fafb2600640 1 -- start start 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 0x7fafac072c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafac1593d0 0x7fafac1f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fafac06e4a0 con 0x7fafac1593d0 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fafac159910 con 0x7fafac152bf0 2024-03-19T03:06:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafb2600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fafac06ea60 con 0x7fafac0c6470 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafab400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 0x7fafac072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafab400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42024/0 (socket says 172.21.15.92:42024) 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafab400640 1 -- 172.21.15.92:0/3078683843 learned_addr learned my addr 172.21.15.92:0/3078683843 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafabe00640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafac1593d0 0x7fafac1f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 -- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 msgr2=0x7fafac0c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 -- 172.21.15.92:0/3078683843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafac1593d0 msgr2=0x7fafac1f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafac1593d0 0x7fafac1f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafaaa00640 1 -- 172.21.15.92:0/3078683843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac073180 con 0x7fafac152bf0 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.333+0000 7fafab400640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafaaa00640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 0x7fafac072c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faf9800c3e0 tx=0x7faf9800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf980074e0 con 0x7fafac152bf0 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf98007680 con 0x7fafac152bf0 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98004a20 con 0x7fafac152bf0 2024-03-19T03:06:07.338 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac0c7940 con 0x7fafac152bf0 2024-03-19T03:06:07.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.334+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac159b10 con 0x7fafac152bf0 2024-03-19T03:06:07.339 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.335+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafac0684c0 con 0x7fafac152bf0 2024-03-19T03:06:07.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.336+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faf98011070 con 0x7fafac152bf0 2024-03-19T03:06:07.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.337+0000 7fafa3400640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 0x7faf8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.337+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7faf9808a750 con 0x7fafac152bf0 2024-03-19T03:06:07.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.337+0000 7fafab400640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 0x7faf8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.338+0000 7fafab400640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 0x7faf8c06e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf9c02b8e0 tx=0x7faf9c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.341+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faf98058990 con 0x7fafac152bf0 2024-03-19T03:06:07.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.540+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafac0c3b50 con 0x7fafac152bf0 2024-03-19T03:06:07.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.542+0000 7fafa3400640 1 -- 172.21.15.92:0/3078683843 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+17584 (secure 0 0 0) 0x7faf9805ca30 con 0x7fafac152bf0 2024-03-19T03:06:07.544 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:07.544 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":38,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T03:05:09.726374+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6839/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6838/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6837/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6838/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6837/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6835/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/1521694689":"2024-03-20T02:57:04.595375+0000","172.21.15.159:6835/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6838/1521694689":"2024-03-20T02:57:04.595375+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:06:07.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.544+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 msgr2=0x7faf8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.544+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 0x7faf8c06e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf9c02b8e0 tx=0x7faf9c008030 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.544+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 msgr2=0x7fafac072c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.544+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 0x7fafac072c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faf9800c3e0 tx=0x7faf9800c8a0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 shutdown_connections 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7faf8c06bbd0 0x7faf8c06e090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafac1593d0 0x7fafac1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafac152bf0 0x7fafac072c40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 --2- 172.21.15.92:0/3078683843 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafac0c6470 0x7fafac0c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 >> 172.21.15.92:0/3078683843 conn(0x7fafac0c8d90 msgr2=0x7fafac06eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:07.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 shutdown_connections 2024-03-19T03:06:07.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.545+0000 7fafb2600640 1 -- 172.21.15.92:0/3078683843 wait complete. 2024-03-19T03:06:07.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:54:49.925691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:54:53.864988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:54:54.662544+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 33, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T03:06:07.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:06:07.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.772+0000 7fb837a00640 1 Processor -- start 2024-03-19T03:06:07.774 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.772+0000 7fb837a00640 1 -- start start 2024-03-19T03:06:07.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.773+0000 7fb837a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb830154dc0 0x7fb830071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.773+0000 7fb837a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb830072380 0x7fb8300c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.773+0000 7fb837a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb83015b740 con 0x7fb8300c60d0 2024-03-19T03:06:07.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.773+0000 7fb837a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb83015ba20 con 0x7fb830154dc0 2024-03-19T03:06:07.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.773+0000 7fb837a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fb8300c8400 con 0x7fb830072380 2024-03-19T03:06:07.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb830154dc0 0x7fb830071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb82fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb8300c60d0 0x7fb830071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54478/0 (socket says 172.21.15.92:54478) 2024-03-19T03:06:07.776 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb82fe00640 1 -- 172.21.15.92:0/1386698564 learned_addr learned my addr 172.21.15.92:0/1386698564 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835a00640 1 --2- 172.21.15.92:0/1386698564 >> v2:172.21.15.159:3300/0 conn(0x7fb830072380 0x7fb8300c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb82f400640 1 -- 172.21.15.92:0/1386698564 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074462124 0 0) 0x7fb83015b740 con 0x7fb8300c60d0 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb82f400640 1 -- 172.21.15.92:0/1386698564 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb818003520 con 0x7fb8300c60d0 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835000640 1 -- 172.21.15.92:0/1386698564 >> v2:172.21.15.159:3300/0 conn(0x7fb830072380 msgr2=0x7fb8300c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835000640 1 --2- 172.21.15.92:0/1386698564 >> v2:172.21.15.159:3300/0 conn(0x7fb830072380 0x7fb8300c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835000640 1 -- 172.21.15.92:0/1386698564 >> v1:172.21.15.159:6789/0 conn(0x7fb8300c60d0 legacy=0x7fb830071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.774+0000 7fb835000640 1 -- 172.21.15.92:0/1386698564 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8300c86e0 con 0x7fb830154dc0 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.775+0000 7fb835000640 1 --2- 172.21.15.92:0/1386698564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb830154dc0 0x7fb830071190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb830075340 tx=0x7fb82402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34be6bbab05ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.775+0000 7fb82f400640 1 -- 172.21.15.92:0/1386698564 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8240315a0 con 0x7fb830154dc0 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.775+0000 7fb82f400640 1 -- 172.21.15.92:0/1386698564 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb824037070 con 0x7fb830154dc0 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.775+0000 7fb82f400640 1 -- 172.21.15.92:0/1386698564 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824031940 con 0x7fb830154dc0 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 -- 172.21.15.92:0/1386698564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb830154dc0 msgr2=0x7fb830071190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 --2- 172.21.15.92:0/1386698564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb830154dc0 0x7fb830071190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb830075340 tx=0x7fb82402b840 comp rx=0 tx=0).stop 2024-03-19T03:06:07.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 -- 172.21.15.92:0/1386698564 shutdown_connections 2024-03-19T03:06:07.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 --2- 172.21.15.92:0/1386698564 >> v2:172.21.15.159:3300/0 conn(0x7fb830072380 0x7fb8300c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 --2- 172.21.15.92:0/1386698564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb830154dc0 0x7fb830071190 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 -- 172.21.15.92:0/1386698564 >> 172.21.15.92:0/1386698564 conn(0x7fb8300c8d90 msgr2=0x7fb8300c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:07.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 -- 172.21.15.92:0/1386698564 shutdown_connections 2024-03-19T03:06:07.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.776+0000 7fb837a00640 1 -- 172.21.15.92:0/1386698564 wait complete. 2024-03-19T03:06:07.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.778+0000 7fb837a00640 1 Processor -- start 2024-03-19T03:06:07.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.778+0000 7fb837a00640 1 -- start start 2024-03-19T03:06:07.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300c60d0 0x7fb830152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb830154dc0 0x7fb830150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb8300c7350 con 0x7fb8300c60d0 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb8300c71d0 con 0x7fb830072380 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb837a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb8300c74d0 con 0x7fb830154dc0 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39532/0 (socket says 172.21.15.92:39532) 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 -- 172.21.15.92:0/3661412314 learned_addr learned my addr 172.21.15.92:0/3661412314 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:07.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb82fe00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300c60d0 0x7fb830152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb830154dc0 0x7fb830150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 -- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb830154dc0 msgr2=0x7fb830150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb830154dc0 0x7fb830150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 -- 172.21.15.92:0/3661412314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300c60d0 msgr2=0x7fb830152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300c60d0 0x7fb830152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.779+0000 7fb835000640 1 -- 172.21.15.92:0/3661412314 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83015e4e0 con 0x7fb830072380 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb835000640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb82402b880 tx=0x7fb8240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824032630 con 0x7fb830072380 2024-03-19T03:06:07.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8240327d0 con 0x7fb830072380 2024-03-19T03:06:07.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83015e770 con 0x7fb830072380 2024-03-19T03:06:07.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824032ba0 con 0x7fb830072380 2024-03-19T03:06:07.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.780+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8301f0700 con 0x7fb830072380 2024-03-19T03:06:07.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.781+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8300c54d0 con 0x7fb830072380 2024-03-19T03:06:07.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.782+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb82404e430 con 0x7fb830072380 2024-03-19T03:06:07.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.783+0000 7fb82cc00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:07.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.783+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fb82402f030 con 0x7fb830072380 2024-03-19T03:06:07.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.783+0000 7fb82fe00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:07.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.784+0000 7fb82fe00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 0x7fb81406e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb830072e10 tx=0x7fb820009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:07.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.787+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb8300c54d0 con 0x7fb830072380 2024-03-19T03:06:07.986 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.984+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8300684c0 con 0x7fb830072380 2024-03-19T03:06:07.987 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.985+0000 7fb82cc00640 1 -- 172.21.15.92:0/3661412314 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7fb824040070 con 0x7fb830072380 2024-03-19T03:06:07.987 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 msgr2=0x7fb81406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 0x7fb81406e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb830072e10 tx=0x7fb820009040 comp rx=0 tx=0).stop 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 msgr2=0x7fb83015dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb82402b880 tx=0x7fb8240046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 shutdown_connections 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb830154dc0 0x7fb830150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300c60d0 0x7fb830152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 --2- 172.21.15.92:0/3661412314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb830072380 0x7fb83015dfa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 >> 172.21.15.92:0/3661412314 conn(0x7fb8300c8d90 msgr2=0x7fb8300c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 shutdown_connections 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:07.988+0000 7fb837a00640 1 -- 172.21.15.92:0/3661412314 wait complete. 2024-03-19T03:06:08.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:06:08.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.214+0000 7f4e39c00640 1 Processor -- start 2024-03-19T03:06:08.216 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.214+0000 7f4e39c00640 1 -- start start 2024-03-19T03:06:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.215+0000 7f4e39c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4e34151ce0 0x7f4e34158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.215+0000 7f4e39c00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4e340c7da0 0x7f4e340c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.215+0000 7f4e39c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e340c3b80 con 0x7f4e34151900 2024-03-19T03:06:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.215+0000 7f4e39c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4e34158860 con 0x7f4e34151ce0 2024-03-19T03:06:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e39c00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f4e34072e70 con 0x7f4e340c7da0 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e32a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4e34151900 0x7f4e340723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53800/0 (socket says 172.21.15.92:53800) 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e32a00640 1 -- 172.21.15.92:0/547379780 learned_addr learned my addr 172.21.15.92:0/547379780 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e33400640 1 --2- 172.21.15.92:0/547379780 >> v2:172.21.15.159:3300/0 conn(0x7f4e340c7da0 0x7f4e340c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e32000640 1 --2- 172.21.15.92:0/547379780 >> v2:172.21.15.92:3300/0 conn(0x7f4e34151ce0 0x7f4e34158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e33400640 1 -- 172.21.15.92:0/547379780 >> v2:172.21.15.92:3300/0 conn(0x7f4e34151ce0 msgr2=0x7f4e34158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e33400640 1 --2- 172.21.15.92:0/547379780 >> v2:172.21.15.92:3300/0 conn(0x7f4e34151ce0 0x7f4e34158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e31600640 1 -- 172.21.15.92:0/547379780 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916769070 0 0) 0x7f4e340c3b80 con 0x7f4e34151900 2024-03-19T03:06:08.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e33400640 1 -- 172.21.15.92:0/547379780 >> v1:172.21.15.92:6789/0 conn(0x7f4e34151900 legacy=0x7f4e340723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.220 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.216+0000 7f4e33400640 1 -- 172.21.15.92:0/547379780 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e34073150 con 0x7f4e340c7da0 2024-03-19T03:06:08.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.218+0000 7f4e33400640 1 --2- 172.21.15.92:0/547379780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e340c7da0 0x7f4e340c81f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e34073700 tx=0x7f4e2c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13a40c9a9fdf09f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.218+0000 7f4e31600640 1 -- 172.21.15.92:0/547379780 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c039070 con 0x7f4e340c7da0 2024-03-19T03:06:08.221 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.218+0000 7f4e31600640 1 -- 172.21.15.92:0/547379780 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e2c02cd00 con 0x7f4e340c7da0 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.218+0000 7f4e31600640 1 -- 172.21.15.92:0/547379780 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c034660 con 0x7f4e340c7da0 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 -- 172.21.15.92:0/547379780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e340c7da0 msgr2=0x7f4e340c81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 --2- 172.21.15.92:0/547379780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e340c7da0 0x7f4e340c81f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e34073700 tx=0x7f4e2c02bf50 comp rx=0 tx=0).stop 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 -- 172.21.15.92:0/547379780 shutdown_connections 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 --2- 172.21.15.92:0/547379780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e340c7da0 0x7f4e340c81f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 --2- 172.21.15.92:0/547379780 >> v2:172.21.15.92:3300/0 conn(0x7f4e34151ce0 0x7f4e34158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 -- 172.21.15.92:0/547379780 >> 172.21.15.92:0/547379780 conn(0x7f4e340c8d90 msgr2=0x7f4e340c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:08.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 -- 172.21.15.92:0/547379780 shutdown_connections 2024-03-19T03:06:08.223 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.219+0000 7f4e39c00640 1 -- 172.21.15.92:0/547379780 wait complete. 2024-03-19T03:06:08.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.221+0000 7f4e39c00640 1 Processor -- start 2024-03-19T03:06:08.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.221+0000 7f4e39c00640 1 -- start start 2024-03-19T03:06:08.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e340c7da0 0x7f4e341f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.225 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e34151ce0 0x7f4e341506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4e340c9af0 con 0x7f4e340c7da0 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4e34071eb0 con 0x7f4e34151ce0 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e39c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4e340c9c70 con 0x7f4e34151900 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e340c7da0 0x7f4e341f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.226 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42056/0 (socket says 172.21.15.92:42056) 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 -- 172.21.15.92:0/969528513 learned_addr learned my addr 172.21.15.92:0/969528513 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e33400640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e34151ce0 0x7f4e341506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 -- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e34151ce0 msgr2=0x7f4e341506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e34151ce0 0x7f4e341506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 -- 172.21.15.92:0/969528513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e340c7da0 msgr2=0x7f4e341f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e340c7da0 0x7f4e341f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.222+0000 7f4e32000640 1 -- 172.21.15.92:0/969528513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e34153890 con 0x7f4e34151900 2024-03-19T03:06:08.227 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e32000640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000e9b0 tx=0x7f4e2000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20017070 con 0x7f4e34151900 2024-03-19T03:06:08.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e2000c9a0 con 0x7f4e34151900 2024-03-19T03:06:08.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000cd90 con 0x7f4e34151900 2024-03-19T03:06:08.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e340c33f0 con 0x7f4e34151900 2024-03-19T03:06:08.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.223+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e34148d30 con 0x7f4e34151900 2024-03-19T03:06:08.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.224+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e340684c0 con 0x7f4e34151900 2024-03-19T03:06:08.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.225+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4e2001a070 con 0x7f4e34151900 2024-03-19T03:06:08.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.226+0000 7f4e2aa00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 0x7f4e1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.226+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f4e2008aff0 con 0x7f4e34151900 2024-03-19T03:06:08.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.226+0000 7f4e32a00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 0x7f4e1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.227+0000 7f4e32a00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 0x7f4e1406e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4e24000f60 tx=0x7f4e24007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.230+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4e34153890 con 0x7f4e34151900 2024-03-19T03:06:08.430 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.429+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e340c4b80 con 0x7f4e34151900 2024-03-19T03:06:08.431 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.429+0000 7f4e2aa00640 1 -- 172.21.15.92:0/969528513 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7f4e200592a0 con 0x7f4e34151900 2024-03-19T03:06:08.431 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 32 2024-03-19T03:06:08.433 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 msgr2=0x7f4e1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 0x7f4e1406e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4e24000f60 tx=0x7f4e24007d50 comp rx=0 tx=0).stop 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 msgr2=0x7f4e34144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000e9b0 tx=0x7f4e2000ee70 comp rx=0 tx=0).stop 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 shutdown_connections 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e1406bbd0 0x7f4e1406e090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e34151ce0 0x7f4e341506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.434 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e34151900 0x7f4e34144e60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 --2- 172.21.15.92:0/969528513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e340c7da0 0x7f4e341f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 >> 172.21.15.92:0/969528513 conn(0x7f4e340c8d90 msgr2=0x7f4e340cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:08.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 shutdown_connections 2024-03-19T03:06:08.435 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.432+0000 7f4e39c00640 1 -- 172.21.15.92:0/969528513 wait complete. 2024-03-19T03:06:08.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:06:08.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.666+0000 7f52c7600640 1 Processor -- start 2024-03-19T03:06:08.668 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.666+0000 7f52c7600640 1 -- start start 2024-03-19T03:06:08.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f52c0152bf0 0x7f52c006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 0x7f52c00c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c01593d0 0x7f52c00c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f52c00c6c90 con 0x7f52c0152bf0 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f52c0159910 con 0x7f52c00c6470 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.667+0000 7f52c7600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f52c006e4a0 con 0x7f52c01593d0 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f52c0152bf0 0x7f52c006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52b7e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 0x7f52c00c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.670 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f52c0152bf0 0x7f52c006eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33934/0 (socket says 172.21.15.92:33934) 2024-03-19T03:06:08.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52b7e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 0x7f52c00c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39552/0 (socket says 172.21.15.92:39552) 2024-03-19T03:06:08.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c5600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c01593d0 0x7f52c00c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 -- 172.21.15.92:0/4098260908 learned_addr learned my addr 172.21.15.92:0/4098260908 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:08.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 -- 172.21.15.92:0/4098260908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c01593d0 msgr2=0x7f52c00c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 --2- 172.21.15.92:0/4098260908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c01593d0 0x7f52c00c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 -- 172.21.15.92:0/4098260908 >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 msgr2=0x7f52c00c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 --2- 172.21.15.92:0/4098260908 >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 0x7f52c00c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.668+0000 7f52c4c00640 1 -- 172.21.15.92:0/4098260908 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00c9a80 con 0x7f52c0152bf0 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.669+0000 7f52c4c00640 1 --2- 172.21.15.92:0/4098260908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c006eef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f52c0075340 tx=0x7f52b002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b318a02d5fa524 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.670+0000 7f52bfe00640 1 -- 172.21.15.92:0/4098260908 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b00317f0 con 0x7f52c0152bf0 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.670+0000 7f52bfe00640 1 -- 172.21.15.92:0/4098260908 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b0037070 con 0x7f52c0152bf0 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.670+0000 7f52bfe00640 1 -- 172.21.15.92:0/4098260908 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0031bb0 con 0x7f52c0152bf0 2024-03-19T03:06:08.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.670+0000 7f52c7600640 1 -- 172.21.15.92:0/4098260908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 msgr2=0x7f52c006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.670+0000 7f52c7600640 1 --2- 172.21.15.92:0/4098260908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c006eef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f52c0075340 tx=0x7f52b002b890 comp rx=0 tx=0).stop 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 -- 172.21.15.92:0/4098260908 shutdown_connections 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 --2- 172.21.15.92:0/4098260908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c01593d0 0x7f52c00c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 --2- 172.21.15.92:0/4098260908 >> v2:172.21.15.159:3300/0 conn(0x7f52c00c6470 0x7f52c00c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 --2- 172.21.15.92:0/4098260908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c006eef0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 -- 172.21.15.92:0/4098260908 >> 172.21.15.92:0/4098260908 conn(0x7f52c00c8d90 msgr2=0x7f52c00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 -- 172.21.15.92:0/4098260908 shutdown_connections 2024-03-19T03:06:08.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.671+0000 7f52c7600640 1 -- 172.21.15.92:0/4098260908 wait complete. 2024-03-19T03:06:08.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.672+0000 7f52c7600640 1 Processor -- start 2024-03-19T03:06:08.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.672+0000 7f52c7600640 1 -- start start 2024-03-19T03:06:08.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 0x7f52c0159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f52c00c8930 con 0x7f52c0152bf0 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f52c006ea60 con 0x7f52c01593d0 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.673+0000 7f52c7600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f52c006e6f0 con 0x7f52c00c6470 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c5600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 0x7f52c0159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42080/0 (socket says 172.21.15.92:42080) 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 -- 172.21.15.92:0/39469142 learned_addr learned my addr 172.21.15.92:0/39469142 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:08.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52b7e00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 -- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 msgr2=0x7f52c0159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 0x7f52c0159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 -- 172.21.15.92:0/39469142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 msgr2=0x7f52c014e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c014e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 -- 172.21.15.92:0/39469142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c01ee930 con 0x7f52c00c6470 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c5600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 0x7f52c0159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.674+0000 7f52c4c00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52b0000c00 tx=0x7f52b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.675+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0032680 con 0x7f52c00c6470 2024-03-19T03:06:08.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.675+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b002e070 con 0x7f52c00c6470 2024-03-19T03:06:08.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.675+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c0149200 con 0x7f52c00c6470 2024-03-19T03:06:08.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.675+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0032a60 con 0x7f52c00c6470 2024-03-19T03:06:08.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.675+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c01eeed0 con 0x7f52c00c6470 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.676+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c00684c0 con 0x7f52c00c6470 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.677+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f52b004c060 con 0x7f52c00c6470 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.678+0000 7f52bd600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.678+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f52b00ac320 con 0x7f52c00c6470 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.678+0000 7f52b7e00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:08.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.678+0000 7f52b7e00640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 0x7f529c06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f52c01e6280 tx=0x7f52a8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:08.688 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.687+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f52b0040070 con 0x7f52c00c6470 2024-03-19T03:06:08.888 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.886+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f52c006f440 con 0x7f52c00c6470 2024-03-19T03:06:08.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.887+0000 7f52bd600640 1 -- 172.21.15.92:0/39469142 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f52b007a5a0 con 0x7f52c00c6470 2024-03-19T03:06:08.889 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 64 2024-03-19T03:06:08.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 msgr2=0x7f529c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 0x7f529c06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f52c01e6280 tx=0x7f52a8009190 comp rx=0 tx=0).stop 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 msgr2=0x7f52c0148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52b0000c00 tx=0x7f52b00046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 shutdown_connections 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52c01593d0 0x7f52c0159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52c0152bf0 0x7f52c014e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 --2- 172.21.15.92:0/39469142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52c00c6470 0x7f52c0148b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:08.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 >> 172.21.15.92:0/39469142 conn(0x7f52c00c8d90 msgr2=0x7f52c0151e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:08.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 shutdown_connections 2024-03-19T03:06:08.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:08.890+0000 7f52c7600640 1 -- 172.21.15.92:0/39469142 wait complete. 2024-03-19T03:06:08.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:06:09.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.121+0000 7f5c71c00640 1 Processor -- start 2024-03-19T03:06:09.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.121+0000 7f5c71c00640 1 -- start start 2024-03-19T03:06:09.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c71c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5c6c150fb0 0x7f5c6c157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:09.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c71c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c158000 con 0x7f5c6c0c6d00 2024-03-19T03:06:09.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c71c00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c0c7220 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c71c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6c0c7500 con 0x7f5c6c150fb0 2024-03-19T03:06:09.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c6a000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5c6c06f7c0 0x7f5c6c0c4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54488/0 (socket says 172.21.15.92:54488) 2024-03-19T03:06:09.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c6a000640 1 -- 172.21.15.92:0/970031345 learned_addr learned my addr 172.21.15.92:0/970031345 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:09.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.123+0000 7f5c6aa00640 1 --2- 172.21.15.92:0/970031345 >> v2:172.21.15.92:3300/0 conn(0x7f5c6c150fb0 0x7f5c6c157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329407354 0 0) 0x7f5c6c0c7220 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c50004630 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c6aa00640 1 -- 172.21.15.92:0/970031345 >> v1:172.21.15.159:6789/0 conn(0x7f5c6c06f7c0 legacy=0x7f5c6c0c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568580706 0 0) 0x7f5c6c158000 con 0x7f5c6c0c6d00 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c6aa00640 1 -- 172.21.15.92:0/970031345 >> v1:172.21.15.92:6789/0 conn(0x7f5c6c0c6d00 legacy=0x7f5c6c06e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.124+0000 7f5c6aa00640 1 -- 172.21.15.92:0/970031345 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c0c77e0 con 0x7f5c6c150fb0 2024-03-19T03:06:09.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.125+0000 7f5c6aa00640 1 --2- 172.21.15.92:0/970031345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c150fb0 0x7f5c6c157ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097a0 tx=0x7f5c5c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3630119ac2e822b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c02f810 con 0x7f5c6c150fb0 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5c02f9b0 con 0x7f5c6c150fb0 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c69600640 1 -- 172.21.15.92:0/970031345 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c02fd60 con 0x7f5c6c150fb0 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c71c00640 1 -- 172.21.15.92:0/970031345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c150fb0 msgr2=0x7f5c6c157ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c71c00640 1 --2- 172.21.15.92:0/970031345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c150fb0 0x7f5c6c157ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097a0 tx=0x7f5c5c02b9f0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c71c00640 1 -- 172.21.15.92:0/970031345 shutdown_connections 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c71c00640 1 --2- 172.21.15.92:0/970031345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c150fb0 0x7f5c6c157ac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.126+0000 7f5c71c00640 1 -- 172.21.15.92:0/970031345 >> 172.21.15.92:0/970031345 conn(0x7f5c6c06b240 msgr2=0x7f5c6c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:09.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.127+0000 7f5c71c00640 1 -- 172.21.15.92:0/970031345 shutdown_connections 2024-03-19T03:06:09.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.127+0000 7f5c71c00640 1 -- 172.21.15.92:0/970031345 wait complete. 2024-03-19T03:06:09.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.128+0000 7f5c71c00640 1 Processor -- start 2024-03-19T03:06:09.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.128+0000 7f5c71c00640 1 -- start start 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c6c0c6d00 0x7f5c6c14d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c6c150fb0 0x7f5c6c1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c8080 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c7da0 con 0x7f5c6c0c6d00 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.129+0000 7f5c71c00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c5310 con 0x7f5c6c150fb0 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:09.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33956/0 (socket says 172.21.15.92:33956) 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 -- 172.21.15.92:0/1382821848 learned_addr learned my addr 172.21.15.92:0/1382821848 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6a000640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c6c0c6d00 0x7f5c6c14d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6b400640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c6c150fb0 0x7f5c6c1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 -- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c6c150fb0 msgr2=0x7f5c6c1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c6c150fb0 0x7f5c6c1472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 -- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c6c0c6d00 msgr2=0x7f5c6c14d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c6c0c6d00 0x7f5c6c14d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 -- 172.21.15.92:0/1382821848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c14dc50 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.130+0000 7f5c6aa00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0098c0 tx=0x7f5c5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.131+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c049070 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.131+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5c03d040 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.131+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c037a50 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.131+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c145a30 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.131+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c145ff0 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.132+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c0684c0 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.133+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5c5c04c070 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.134+0000 7f5c62a00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 0x7f5c4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:09.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.134+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f5c5c0acd30 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.134+0000 7f5c6a000640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 0x7f5c4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:09.137 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.135+0000 7f5c6a000640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 0x7f5c4806e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c1e8cb0 tx=0x7f5c54007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:09.140 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.139+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5c5c033030 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.350+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5c6c154230 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.351+0000 7f5c62a00640 1 -- 172.21.15.92:0/1382821848 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v38) ==== 81+0+620 (secure 0 0 0) 0x7f5c5c07af80 con 0x7f5c6c06f7c0 2024-03-19T03:06:09.353 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6839/3845908330 2024-03-20T03:05:09.726289+0000 2024-03-19T03:06:09.353 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6838/3845908330 2024-03-20T03:05:09.726289+0000 2024-03-19T03:06:09.353 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6837/2066707194 2024-03-20T03:03:24.698071+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6839/2444776025 2024-03-20T03:00:59.655927+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6838/2444776025 2024-03-20T03:00:59.655927+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6837/2387435110 2024-03-20T02:59:24.630258+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6835/2066707194 2024-03-20T03:03:24.698071+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6839/1521694689 2024-03-20T02:57:04.595375+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6835/2387435110 2024-03-20T02:59:24.630258+0000 2024-03-19T03:06:09.354 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.159:6838/1521694689 2024-03-20T02:57:04.595375+0000 2024-03-19T03:06:09.355 INFO:teuthology.orchestra.run.smithi092.stderr:listed 10 entries 2024-03-19T03:06:09.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 msgr2=0x7f5c4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 0x7f5c4806e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c1e8cb0 tx=0x7f5c54007d30 comp rx=0 tx=0).stop 2024-03-19T03:06:09.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 msgr2=0x7f5c6c1eedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0098c0 tx=0x7f5c5c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 shutdown_connections 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f5c4806bbd0 0x7f5c4806e090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c6c150fb0 0x7f5c6c1472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c6c0c6d00 0x7f5c6c14d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 --2- 172.21.15.92:0/1382821848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c6c06f7c0 0x7f5c6c1eedf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.354+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 >> 172.21.15.92:0/1382821848 conn(0x7f5c6c06b240 msgr2=0x7f5c6c14a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.355+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 shutdown_connections 2024-03-19T03:06:09.358 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:09.355+0000 7f5c71c00640 1 -- 172.21.15.92:0/1382821848 wait complete. 2024-03-19T03:06:09.374 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-19T03:06:09.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:06:09.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-19T03:06:12.996 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:06:12.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:06:12.997 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T03:06:13.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:06:13.048 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-19T03:06:13.143 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T03:06:13.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:06:13.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T03:06:13.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:06:13.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link delete ceph-brx 2024-03-19T03:06:13.371 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-19T03:06:13.380 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T03:06:13.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.566+0000 7f2081200640 1 Processor -- start 2024-03-19T03:06:13.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.566+0000 7f2081200640 1 -- start start 2024-03-19T03:06:13.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2081200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2081200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f207c0c5f50 0x7f207c152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2081200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f207c0c6d00 con 0x7f207c06d160 2024-03-19T03:06:13.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2081200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f207c06df20 con 0x7f207c0c5f50 2024-03-19T03:06:13.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2081200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f207c06d610 con 0x7f207c0c5b70 2024-03-19T03:06:13.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f207b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2072a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f207c0c5f50 0x7f207c152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2072a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f207c0c5f50 0x7f207c152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39592/0 (socket says 172.21.15.92:39592) 2024-03-19T03:06:13.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f207b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42100/0 (socket says 172.21.15.92:42100) 2024-03-19T03:06:13.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.568+0000 7f2072a00640 1 -- 172.21.15.92:0/110584498 learned_addr learned my addr 172.21.15.92:0/110584498 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:13.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.569+0000 7f2072a00640 1 -- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 msgr2=0x7f207c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:13.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.569+0000 7f2072a00640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.569+0000 7f2072a00640 1 -- 172.21.15.92:0/110584498 >> v1:172.21.15.92:6789/0 conn(0x7f207c06d160 legacy=0x7f207c152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.569+0000 7f2072a00640 1 -- 172.21.15.92:0/110584498 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c0c6fa0 con 0x7f207c0c5f50 2024-03-19T03:06:13.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.569+0000 7f207b400640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:13.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.570+0000 7f2072a00640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c0c5f50 0x7f207c152520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2064009870 tx=0x7f206402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b6402ddd8612e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.570+0000 7f207aa00640 1 -- 172.21.15.92:0/110584498 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206402f830 con 0x7f207c0c5f50 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.570+0000 7f207aa00640 1 -- 172.21.15.92:0/110584498 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206402f9d0 con 0x7f207c0c5f50 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.570+0000 7f207aa00640 1 -- 172.21.15.92:0/110584498 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206402fda0 con 0x7f207c0c5f50 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 -- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c0c5f50 msgr2=0x7f207c152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c0c5f50 0x7f207c152520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2064009870 tx=0x7f206402ba10 comp rx=0 tx=0).stop 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 -- 172.21.15.92:0/110584498 shutdown_connections 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c0c5f50 0x7f207c152520 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 --2- 172.21.15.92:0/110584498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 -- 172.21.15.92:0/110584498 >> 172.21.15.92:0/110584498 conn(0x7f207c0cc1f0 msgr2=0x7f207c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:13.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 -- 172.21.15.92:0/110584498 shutdown_connections 2024-03-19T03:06:13.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.571+0000 7f2081200640 1 -- 172.21.15.92:0/110584498 wait complete. 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.573+0000 7f2081200640 1 Processor -- start 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.573+0000 7f2081200640 1 -- start start 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f207c0c5f50 0x7f207c151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f207c0729b0 con 0x7f207c0c5f50 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f207c06dbb0 con 0x7f207c06d160 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2081200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f207c06e3b0 con 0x7f207c0c5b70 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2072a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39596/0 (socket says 172.21.15.92:39596) 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 -- 172.21.15.92:0/3141853183 learned_addr learned my addr 172.21.15.92:0/3141853183 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207be00640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f207c0c5f50 0x7f207c151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 -- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 msgr2=0x7f207c1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c1f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 -- 172.21.15.92:0/3141853183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f207c0c5f50 msgr2=0x7f207c151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f207c0c5f50 0x7f207c151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f207b400640 1 -- 172.21.15.92:0/3141853183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c1551b0 con 0x7f207c06d160 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.574+0000 7f2072a00640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f207b400640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f206c00ec10 tx=0x7f206c00c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c017070 con 0x7f207c06d160 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206c00cc30 con 0x7f207c06d160 2024-03-19T03:06:13.578 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c013470 con 0x7f207c06d160 2024-03-19T03:06:13.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207c155490 con 0x7f207c06d160 2024-03-19T03:06:13.579 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.575+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f207c1f0680 con 0x7f207c06d160 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.576+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f207c04f580 con 0x7f207c06d160 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.577+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f206c01a070 con 0x7f207c06d160 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.578+0000 7f2073e00640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 0x7f205806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.578+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f206c08b0a0 con 0x7f207c06d160 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.578+0000 7f2072a00640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 0x7f205806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.579+0000 7f2072a00640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 0x7f205806e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2064036000 tx=0x7f2064008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:13.582 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.582+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f206c0591d0 con 0x7f207c06d160 2024-03-19T03:06:13.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.779+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f207c0c68a0 con 0x7f205806bbd0 2024-03-19T03:06:13.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.787+0000 7f2073e00640 1 -- 172.21.15.92:0/3141853183 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256757 (secure 0 0 0) 0x7f207c0c68a0 con 0x7f205806bbd0 2024-03-19T03:06:13.791 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:13.792 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T03:06:13.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 msgr2=0x7f205806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 0x7f205806e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2064036000 tx=0x7f2064008030 comp rx=0 tx=0).stop 2024-03-19T03:06:13.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 msgr2=0x7f207c1f1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:13.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f206c00ec10 tx=0x7f206c00c520 comp rx=0 tx=0).stop 2024-03-19T03:06:13.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 shutdown_connections 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f205806bbd0 0x7f205806e090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f207c0c5f50 0x7f207c151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f207c0c5b70 0x7f207c1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 --2- 172.21.15.92:0/3141853183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f207c06d160 0x7f207c1f1160 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 >> 172.21.15.92:0/3141853183 conn(0x7f207c0cc1f0 msgr2=0x7f207c0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 shutdown_connections 2024-03-19T03:06:13.796 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:13.792+0000 7f2081200640 1 -- 172.21.15.92:0/3141853183 wait complete. 2024-03-19T03:06:13.812 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-19T03:06:11.973824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1167076772,"num_objects":10705,"num_object_clones":0,"num_object_copies":21410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10705,"num_whiteouts":0,"num_read":2482,"num_read_kb":1090930,"num_write":33162,"num_write_kb":1297086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32844,"ondisk_log_size":32844,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2354904,"kb_used_data":1776536,"kb_used_omap":12,"kb_used_meta":578291,"kb_avail":752619816,"statfs":{"total":773094113280,"available":770682691584,"internally_reserved":0,"allocated":1819172864,"data_stored":2309608307,"data_compressed":213046118,"data_compressed_allocated":243834880,"data_compressed_original":787468288,"omap_allocated":12750,"internal_metadata":592170546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35751009,"num_objects":1891,"num_object_clones":0,"num_object_copies":3782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":36304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"3.3f","version":"38'340","reported_seq":2442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.831501+0000","last_change":"2024-03-19T03:00:04.520460+0000","last_active":"2024-03-19T03:06:09.831501+0000","last_peered":"2024-03-19T03:06:09.831501+0000","last_clean":"2024-03-19T03:06:09.831501+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T03:06:09.831501+0000","last_undegraded":"2024-03-19T03:06:09.831501+0000","last_fullsized":"2024-03-19T03:06:09.831501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:33.132110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045874986,"stat_sum":{"num_bytes":26339349,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":26,"num_read_kb":8207,"num_write":340,"num_write_kb":25903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"38'315","reported_seq":2418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.839204+0000","last_change":"2024-03-19T03:00:04.730981+0000","last_active":"2024-03-19T03:06:09.839204+0000","last_peered":"2024-03-19T03:06:09.839204+0000","last_clean":"2024-03-19T03:06:09.839204+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T03:06:09.839204+0000","last_undegraded":"2024-03-19T03:06:09.839204+0000","last_fullsized":"2024-03-19T03:06:09.839204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:40.160269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047342589999999997,"stat_sum":{"num_bytes":1430220,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":315,"num_write_kb":1566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"38'322","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.810784+0000","last_change":"2024-03-19T03:00:04.294476+0000","last_active":"2024-03-19T03:06:09.810784+0000","last_peered":"2024-03-19T03:06:09.810784+0000","last_clean":"2024-03-19T03:06:09.810784+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T03:06:09.810784+0000","last_undegraded":"2024-03-19T03:06:09.810784+0000","last_fullsized":"2024-03-19T03:06:09.810784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:54:31.834322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069861674999999998,"stat_sum":{"num_bytes":15481473,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":33,"num_read_kb":8211,"num_write":321,"num_write_kb":23482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"38'326","reported_seq":2421,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813773+0000","last_change":"2024-03-19T03:00:04.461050+0000","last_active":"2024-03-19T03:06:09.813773+0000","last_peered":"2024-03-19T03:06:09.813773+0000","last_clean":"2024-03-19T03:06:09.813773+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T03:06:09.813773+0000","last_undegraded":"2024-03-19T03:06:09.813773+0000","last_fullsized":"2024-03-19T03:06:09.813773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:49.707937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054309962000000003,"stat_sum":{"num_bytes":20068260,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":19,"num_read_kb":8201,"num_write":329,"num_write_kb":19659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"38'348","reported_seq":2448,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813079+0000","last_change":"2024-03-19T03:00:04.518539+0000","last_active":"2024-03-19T03:06:09.813079+0000","last_peered":"2024-03-19T03:06:09.813079+0000","last_clean":"2024-03-19T03:06:09.813079+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T03:06:09.813079+0000","last_undegraded":"2024-03-19T03:06:09.813079+0000","last_fullsized":"2024-03-19T03:06:09.813079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:46.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051753202999999998,"stat_sum":{"num_bytes":6591355,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":25,"num_read_kb":4114,"num_write":347,"num_write_kb":6624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"38'331","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823940+0000","last_change":"2024-03-19T03:00:04.458228+0000","last_active":"2024-03-19T03:06:09.823940+0000","last_peered":"2024-03-19T03:06:09.823940+0000","last_clean":"2024-03-19T03:06:09.823940+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T03:06:09.823940+0000","last_undegraded":"2024-03-19T03:06:09.823940+0000","last_fullsized":"2024-03-19T03:06:09.823940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:05:00.342939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051735362,"stat_sum":{"num_bytes":12769638,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":28,"num_read_kb":4115,"num_write":331,"num_write_kb":12643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"38'343","reported_seq":2440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.849648+0000","last_change":"2024-03-19T03:00:04.518609+0000","last_active":"2024-03-19T03:06:09.849648+0000","last_peered":"2024-03-19T03:06:09.849648+0000","last_clean":"2024-03-19T03:06:09.849648+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T03:06:09.849648+0000","last_undegraded":"2024-03-19T03:06:09.849648+0000","last_fullsized":"2024-03-19T03:06:09.849648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:41.608998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051913059999999997,"stat_sum":{"num_bytes":23638346,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":344,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"38'298","reported_seq":2388,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.795953+0000","last_change":"2024-03-19T03:00:04.268010+0000","last_active":"2024-03-19T03:06:09.795953+0000","last_peered":"2024-03-19T03:06:09.795953+0000","last_clean":"2024-03-19T03:06:09.795953+0000","last_became_active":"2024-03-19T02:55:45.392555+0000","last_became_peered":"2024-03-19T02:55:45.392555+0000","last_unstale":"2024-03-19T03:06:09.795953+0000","last_undegraded":"2024-03-19T03:06:09.795953+0000","last_fullsized":"2024-03-19T03:06:09.795953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:37.417889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052882827,"stat_sum":{"num_bytes":15124789,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":21,"num_read_kb":4106,"num_write":297,"num_write_kb":14929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"38'322","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828302+0000","last_change":"2024-03-19T03:00:04.520474+0000","last_active":"2024-03-19T03:06:09.828302+0000","last_peered":"2024-03-19T03:06:09.828302+0000","last_clean":"2024-03-19T03:06:09.828302+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T03:06:09.828302+0000","last_undegraded":"2024-03-19T03:06:09.828302+0000","last_fullsized":"2024-03-19T03:06:09.828302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:29.088176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045138699999999997,"stat_sum":{"num_bytes":8352128,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":30,"num_read_kb":4119,"num_write":322,"num_write_kb":12425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"38'304","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.757119+0000","last_change":"2024-03-19T03:00:04.469206+0000","last_active":"2024-03-19T03:06:09.757119+0000","last_peered":"2024-03-19T03:06:09.757119+0000","last_clean":"2024-03-19T03:06:09.757119+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T03:06:09.757119+0000","last_undegraded":"2024-03-19T03:06:09.757119+0000","last_fullsized":"2024-03-19T03:06:09.757119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.863780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033183580999999997,"stat_sum":{"num_bytes":17872088,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":8208,"num_write":302,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"38'305","reported_seq":2406,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.768960+0000","last_change":"2024-03-19T03:00:04.468803+0000","last_active":"2024-03-19T03:06:09.768960+0000","last_peered":"2024-03-19T03:06:09.768960+0000","last_clean":"2024-03-19T03:06:09.768960+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T03:06:09.768960+0000","last_undegraded":"2024-03-19T03:06:09.768960+0000","last_fullsized":"2024-03-19T03:06:09.768960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:05.106811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036038919000000003,"stat_sum":{"num_bytes":18621884,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":26,"num_read_kb":8210,"num_write":305,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"38'292","reported_seq":2385,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.761101+0000","last_change":"2024-03-19T03:00:04.233381+0000","last_active":"2024-03-19T03:06:09.761101+0000","last_peered":"2024-03-19T03:06:09.761101+0000","last_clean":"2024-03-19T03:06:09.761101+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T03:06:09.761101+0000","last_undegraded":"2024-03-19T03:06:09.761101+0000","last_fullsized":"2024-03-19T03:06:09.761101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:36:06.236162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016224016000000001,"stat_sum":{"num_bytes":10799086,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":18,"num_read_kb":10,"num_write":291,"num_write_kb":14795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"38'286","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.918837+0000","last_change":"2024-03-19T03:00:04.638962+0000","last_active":"2024-03-19T03:06:09.918837+0000","last_peered":"2024-03-19T03:06:09.918837+0000","last_clean":"2024-03-19T03:06:09.918837+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T03:06:09.918837+0000","last_undegraded":"2024-03-19T03:06:09.918837+0000","last_fullsized":"2024-03-19T03:06:09.918837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:12.950806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061626093999999999,"stat_sum":{"num_bytes":18869857,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":26,"num_read_kb":4115,"num_write":285,"num_write_kb":26775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"38'330","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.890035+0000","last_change":"2024-03-19T03:00:04.294731+0000","last_active":"2024-03-19T03:06:09.890035+0000","last_peered":"2024-03-19T03:06:09.890035+0000","last_clean":"2024-03-19T03:06:09.890035+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.890035+0000","last_undegraded":"2024-03-19T03:06:09.890035+0000","last_fullsized":"2024-03-19T03:06:09.890035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.987902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.06936341,"stat_sum":{"num_bytes":7021822,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":330,"num_write_kb":6980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"38'351","reported_seq":2453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.838055+0000","last_change":"2024-03-19T03:00:04.731871+0000","last_active":"2024-03-19T03:06:09.838055+0000","last_peered":"2024-03-19T03:06:09.838055+0000","last_clean":"2024-03-19T03:06:09.838055+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T03:06:09.838055+0000","last_undegraded":"2024-03-19T03:06:09.838055+0000","last_fullsized":"2024-03-19T03:06:09.838055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:26.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047989829999999997,"stat_sum":{"num_bytes":22976920,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":351,"num_write_kb":22623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"38'348","reported_seq":2450,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.868400+0000","last_change":"2024-03-19T03:00:04.463722+0000","last_active":"2024-03-19T03:06:09.868400+0000","last_peered":"2024-03-19T03:06:09.868400+0000","last_clean":"2024-03-19T03:06:09.868400+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T03:06:09.868400+0000","last_undegraded":"2024-03-19T03:06:09.868400+0000","last_fullsized":"2024-03-19T03:06:09.868400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:06.211614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054164370000000003,"stat_sum":{"num_bytes":10727184,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":27,"num_read_kb":4115,"num_write":346,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"38'344","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.771765+0000","last_change":"2024-03-19T03:00:04.463595+0000","last_active":"2024-03-19T03:06:09.771765+0000","last_peered":"2024-03-19T03:06:09.771765+0000","last_clean":"2024-03-19T03:06:09.771765+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T03:06:09.771765+0000","last_undegraded":"2024-03-19T03:06:09.771765+0000","last_fullsized":"2024-03-19T03:06:09.771765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:53:29.237872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054055032000000003,"stat_sum":{"num_bytes":10124226,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":32,"num_read_kb":4117,"num_write":343,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"38'290","reported_seq":2398,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798540+0000","last_change":"2024-03-19T03:00:04.265271+0000","last_active":"2024-03-19T03:06:09.798540+0000","last_peered":"2024-03-19T03:06:09.798540+0000","last_clean":"2024-03-19T03:06:09.798540+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T03:06:09.798540+0000","last_undegraded":"2024-03-19T03:06:09.798540+0000","last_fullsized":"2024-03-19T03:06:09.798540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:17.102471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.049230071,"stat_sum":{"num_bytes":7542726,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":289,"num_write_kb":7514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"38'322","reported_seq":2430,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828563+0000","last_change":"2024-03-19T03:00:04.518999+0000","last_active":"2024-03-19T03:06:09.828563+0000","last_peered":"2024-03-19T03:06:09.828563+0000","last_clean":"2024-03-19T03:06:09.828563+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T03:06:09.828563+0000","last_undegraded":"2024-03-19T03:06:09.828563+0000","last_fullsized":"2024-03-19T03:06:09.828563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:30.582720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050023194999999999,"stat_sum":{"num_bytes":14383098,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":32,"num_read_kb":16400,"num_write":321,"num_write_kb":18311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"38'352","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.764360+0000","last_change":"2024-03-19T03:00:04.469246+0000","last_active":"2024-03-19T03:06:09.764360+0000","last_peered":"2024-03-19T03:06:09.764360+0000","last_clean":"2024-03-19T03:06:09.764360+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T03:06:09.764360+0000","last_undegraded":"2024-03-19T03:06:09.764360+0000","last_fullsized":"2024-03-19T03:06:09.764360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:17:26.060817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033531875000000003,"stat_sum":{"num_bytes":23233989,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":33,"num_read_kb":12308,"num_write":352,"num_write_kb":31057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"38'318","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925503+0000","last_change":"2024-03-19T03:00:04.294471+0000","last_active":"2024-03-19T03:06:09.925503+0000","last_peered":"2024-03-19T03:06:09.925503+0000","last_clean":"2024-03-19T03:06:09.925503+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T03:06:09.925503+0000","last_undegraded":"2024-03-19T03:06:09.925503+0000","last_fullsized":"2024-03-19T03:06:09.925503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:35.963767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068442499000000004,"stat_sum":{"num_bytes":13700535,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":33,"num_read_kb":3602,"num_write":320,"num_write_kb":13502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"38'341","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806378+0000","last_change":"2024-03-19T03:00:04.460208+0000","last_active":"2024-03-19T03:06:09.806378+0000","last_peered":"2024-03-19T03:06:09.806378+0000","last_clean":"2024-03-19T03:06:09.806378+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T03:06:09.806378+0000","last_undegraded":"2024-03-19T03:06:09.806378+0000","last_fullsized":"2024-03-19T03:06:09.806378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:08.718707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053636608000000002,"stat_sum":{"num_bytes":9951503,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":340,"num_write_kb":13990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"38'325","reported_seq":2437,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.900508+0000","last_change":"2024-03-19T03:00:04.634734+0000","last_active":"2024-03-19T03:06:09.900508+0000","last_peered":"2024-03-19T03:06:09.900508+0000","last_clean":"2024-03-19T03:06:09.900508+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T03:06:09.900508+0000","last_undegraded":"2024-03-19T03:06:09.900508+0000","last_fullsized":"2024-03-19T03:06:09.900508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:56.724601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069022274999999994,"stat_sum":{"num_bytes":15469691,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":37,"num_read_kb":13386,"num_write":325,"num_write_kb":19374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"38'347","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.784715+0000","last_change":"2024-03-19T03:00:04.464949+0000","last_active":"2024-03-19T03:06:09.784715+0000","last_peered":"2024-03-19T03:06:09.784715+0000","last_clean":"2024-03-19T03:06:09.784715+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T03:06:09.784715+0000","last_undegraded":"2024-03-19T03:06:09.784715+0000","last_fullsized":"2024-03-19T03:06:09.784715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:40.894087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055361406000000002,"stat_sum":{"num_bytes":13836038,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":38,"num_read_kb":4325,"num_write":347,"num_write_kb":13690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"38'316","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896418+0000","last_change":"2024-03-19T03:00:04.636666+0000","last_active":"2024-03-19T03:06:09.896418+0000","last_peered":"2024-03-19T03:06:09.896418+0000","last_clean":"2024-03-19T03:06:09.896418+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T03:06:09.896418+0000","last_undegraded":"2024-03-19T03:06:09.896418+0000","last_fullsized":"2024-03-19T03:06:09.896418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:13:09.492010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066517528000000006,"stat_sum":{"num_bytes":14678298,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":314,"num_write_kb":14504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"38'357","reported_seq":2459,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.763988+0000","last_change":"2024-03-19T03:00:04.469451+0000","last_active":"2024-03-19T03:06:09.763988+0000","last_peered":"2024-03-19T03:06:09.763988+0000","last_clean":"2024-03-19T03:06:09.763988+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T03:06:09.763988+0000","last_undegraded":"2024-03-19T03:06:09.763988+0000","last_fullsized":"2024-03-19T03:06:09.763988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:08.415495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033380219000000003,"stat_sum":{"num_bytes":6267254,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":357,"num_write_kb":6310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"38'323","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.797829+0000","last_change":"2024-03-19T03:00:04.731007+0000","last_active":"2024-03-19T03:06:09.797829+0000","last_peered":"2024-03-19T03:06:09.797829+0000","last_clean":"2024-03-19T03:06:09.797829+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T03:06:09.797829+0000","last_undegraded":"2024-03-19T03:06:09.797829+0000","last_fullsized":"2024-03-19T03:06:09.797829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:38.782912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046987292,"stat_sum":{"num_bytes":20194837,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":27,"num_read_kb":4111,"num_write":325,"num_write_kb":23913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"38'312","reported_seq":2410,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925331+0000","last_change":"2024-03-19T03:00:04.639714+0000","last_active":"2024-03-19T03:06:09.925331+0000","last_peered":"2024-03-19T03:06:09.925331+0000","last_clean":"2024-03-19T03:06:09.925331+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T03:06:09.925331+0000","last_undegraded":"2024-03-19T03:06:09.925331+0000","last_fullsized":"2024-03-19T03:06:09.925331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:54.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054969302999999997,"stat_sum":{"num_bytes":19046740,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":22,"num_read_kb":7383,"num_write":312,"num_write_kb":22038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"38'348","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.926885+0000","last_change":"2024-03-19T03:00:04.295066+0000","last_active":"2024-03-19T03:06:09.926885+0000","last_peered":"2024-03-19T03:06:09.926885+0000","last_clean":"2024-03-19T03:06:09.926885+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T03:06:09.926885+0000","last_undegraded":"2024-03-19T03:06:09.926885+0000","last_fullsized":"2024-03-19T03:06:09.926885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:51.825764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068635079000000002,"stat_sum":{"num_bytes":22225330,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":36,"num_read_kb":12306,"num_write":348,"num_write_kb":21889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"38'345","reported_seq":2457,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781725+0000","last_change":"2024-03-19T03:00:04.460792+0000","last_active":"2024-03-19T03:06:09.781725+0000","last_peered":"2024-03-19T03:06:09.781725+0000","last_clean":"2024-03-19T03:06:09.781725+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T03:06:09.781725+0000","last_undegraded":"2024-03-19T03:06:09.781725+0000","last_fullsized":"2024-03-19T03:06:09.781725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:11.679527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054050365000000003,"stat_sum":{"num_bytes":13922214,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":36,"num_read_kb":4118,"num_write":345,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"38'394","reported_seq":2502,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.805290+0000","last_change":"2024-03-19T03:00:04.266403+0000","last_active":"2024-03-19T03:06:09.805290+0000","last_peered":"2024-03-19T03:06:09.805290+0000","last_clean":"2024-03-19T03:06:09.805290+0000","last_became_active":"2024-03-19T02:55:45.950634+0000","last_became_peered":"2024-03-19T02:55:45.950634+0000","last_unstale":"2024-03-19T03:06:09.805290+0000","last_undegraded":"2024-03-19T03:06:09.805290+0000","last_fullsized":"2024-03-19T03:06:09.805290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:59.262092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050892488,"stat_sum":{"num_bytes":11658804,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":27,"num_read_kb":4113,"num_write":395,"num_write_kb":15629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"38'360","reported_seq":2472,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.864349+0000","last_change":"2024-03-19T03:00:04.467477+0000","last_active":"2024-03-19T03:06:09.864349+0000","last_peered":"2024-03-19T03:06:09.864349+0000","last_clean":"2024-03-19T03:06:09.864349+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T03:06:09.864349+0000","last_undegraded":"2024-03-19T03:06:09.864349+0000","last_fullsized":"2024-03-19T03:06:09.864349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:19.601901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035464137999999999,"stat_sum":{"num_bytes":11414841,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":36,"num_read_kb":4123,"num_write":359,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"38'322","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781195+0000","last_change":"2024-03-19T03:00:04.233486+0000","last_active":"2024-03-19T03:06:09.781195+0000","last_peered":"2024-03-19T03:06:09.781195+0000","last_clean":"2024-03-19T03:06:09.781195+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T03:06:09.781195+0000","last_undegraded":"2024-03-19T03:06:09.781195+0000","last_fullsized":"2024-03-19T03:06:09.781195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.162762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016612378000000001,"stat_sum":{"num_bytes":15052974,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":23,"num_read_kb":4112,"num_write":322,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'49","reported_seq":112,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.737087+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T03:05:09.737087+0000","last_peered":"2024-03-19T03:05:09.737087+0000","last_clean":"2024-03-19T03:05:09.737087+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T03:05:09.737087+0000","last_undegraded":"2024-03-19T03:05:09.737087+0000","last_fullsized":"2024-03-19T03:05:09.737087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":52,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"38'306","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.922358+0000","last_change":"2024-03-19T03:00:04.295011+0000","last_active":"2024-03-19T03:06:09.922358+0000","last_peered":"2024-03-19T03:06:09.922358+0000","last_clean":"2024-03-19T03:06:09.922358+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T03:06:09.922358+0000","last_undegraded":"2024-03-19T03:06:09.922358+0000","last_fullsized":"2024-03-19T03:06:09.922358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:17.919923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066278250999999996,"stat_sum":{"num_bytes":6674098,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":15,"num_write":306,"num_write_kb":6677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"38'198","reported_seq":268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.188393+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T03:05:17.188393+0000","last_peered":"2024-03-19T03:05:17.188393+0000","last_clean":"2024-03-19T03:05:17.188393+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T03:05:17.188393+0000","last_undegraded":"2024-03-19T03:05:17.188393+0000","last_fullsized":"2024-03-19T03:05:17.188393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":61440,"num_write":198,"num_write_kb":16593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"38'29","reported_seq":87,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.904891+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T03:05:15.904891+0000","last_peered":"2024-03-19T03:05:15.904891+0000","last_clean":"2024-03-19T03:05:15.904891+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T03:05:15.904891+0000","last_undegraded":"2024-03-19T03:05:15.904891+0000","last_fullsized":"2024-03-19T03:05:15.904891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":29,"num_write_kb":4131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"38'386","reported_seq":2495,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.946964+0000","last_change":"2024-03-19T03:00:04.638741+0000","last_active":"2024-03-19T03:06:09.946964+0000","last_peered":"2024-03-19T03:06:09.946964+0000","last_clean":"2024-03-19T03:06:09.946964+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T03:06:09.946964+0000","last_undegraded":"2024-03-19T03:06:09.946964+0000","last_fullsized":"2024-03-19T03:06:09.946964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:31.894668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061005943,"stat_sum":{"num_bytes":22416031,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":34,"num_read_kb":16407,"num_write":386,"num_write_kb":31278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"38'751","reported_seq":814,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:06.635700+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T03:06:06.635700+0000","last_peered":"2024-03-19T03:06:06.635700+0000","last_clean":"2024-03-19T03:06:06.635700+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T03:06:06.635700+0000","last_undegraded":"2024-03-19T03:06:06.635700+0000","last_fullsized":"2024-03-19T03:06:06.635700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":748,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"38'321","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828476+0000","last_change":"2024-03-19T03:00:04.731670+0000","last_active":"2024-03-19T03:06:09.828476+0000","last_peered":"2024-03-19T03:06:09.828476+0000","last_clean":"2024-03-19T03:06:09.828476+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T03:06:09.828476+0000","last_undegraded":"2024-03-19T03:06:09.828476+0000","last_fullsized":"2024-03-19T03:06:09.828476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:37.837633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048289845999999997,"stat_sum":{"num_bytes":20433584,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":31,"num_read_kb":16402,"num_write":319,"num_write_kb":24181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'8","reported_seq":62,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.733040+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T03:05:09.733040+0000","last_peered":"2024-03-19T03:05:09.733040+0000","last_clean":"2024-03-19T03:05:09.733040+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T03:05:09.733040+0000","last_undegraded":"2024-03-19T03:05:09.733040+0000","last_fullsized":"2024-03-19T03:05:09.733040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"38'348","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.804516+0000","last_change":"2024-03-19T03:00:04.266848+0000","last_active":"2024-03-19T03:06:09.804516+0000","last_peered":"2024-03-19T03:06:09.804516+0000","last_clean":"2024-03-19T03:06:09.804516+0000","last_became_active":"2024-03-19T02:55:45.403591+0000","last_became_peered":"2024-03-19T02:55:45.403591+0000","last_unstale":"2024-03-19T03:06:09.804516+0000","last_undegraded":"2024-03-19T03:06:09.804516+0000","last_fullsized":"2024-03-19T03:06:09.804516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:03.822452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050955334999999997,"stat_sum":{"num_bytes":22779940,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":32,"num_read_kb":10328,"num_write":348,"num_write_kb":22426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"38'1621","reported_seq":1681,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.125122+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T03:05:54.125122+0000","last_peered":"2024-03-19T03:05:54.125122+0000","last_clean":"2024-03-19T03:05:54.125122+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T03:05:54.125122+0000","last_undegraded":"2024-03-19T03:05:54.125122+0000","last_fullsized":"2024-03-19T03:05:54.125122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":19565,"num_write":1617,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"38'308","reported_seq":2416,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.889708+0000","last_change":"2024-03-19T03:00:04.295350+0000","last_active":"2024-03-19T03:06:09.889708+0000","last_peered":"2024-03-19T03:06:09.889708+0000","last_clean":"2024-03-19T03:06:09.889708+0000","last_became_active":"2024-03-19T02:55:45.952615+0000","last_became_peered":"2024-03-19T02:55:45.952615+0000","last_unstale":"2024-03-19T03:06:09.889708+0000","last_undegraded":"2024-03-19T03:06:09.889708+0000","last_fullsized":"2024-03-19T03:06:09.889708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:35:42.703979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067870387000000004,"stat_sum":{"num_bytes":9678845,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":28,"num_read_kb":4114,"num_write":307,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"38'258","reported_seq":327,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:39.898558+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T03:05:39.898558+0000","last_peered":"2024-03-19T03:05:39.898558+0000","last_clean":"2024-03-19T03:05:39.898558+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T03:05:39.898558+0000","last_undegraded":"2024-03-19T03:05:39.898558+0000","last_fullsized":"2024-03-19T03:05:39.898558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":16407,"num_write":255,"num_write_kb":8344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"38'309","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.759258+0000","last_change":"2024-03-19T03:00:04.233584+0000","last_active":"2024-03-19T03:06:09.759258+0000","last_peered":"2024-03-19T03:06:09.759258+0000","last_clean":"2024-03-19T03:06:09.759258+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T03:06:09.759258+0000","last_undegraded":"2024-03-19T03:06:09.759258+0000","last_fullsized":"2024-03-19T03:06:09.759258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:28.958102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016376714000000001,"stat_sum":{"num_bytes":15448536,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":23,"num_read_kb":4110,"num_write":308,"num_write_kb":19342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"38'161","reported_seq":245,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.120316+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T03:05:54.120316+0000","last_peered":"2024-03-19T03:05:54.120316+0000","last_clean":"2024-03-19T03:05:54.120316+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T03:05:54.120316+0000","last_undegraded":"2024-03-19T03:05:54.120316+0000","last_fullsized":"2024-03-19T03:05:54.120316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":57372,"num_write":154,"num_write_kb":12387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"38'321","reported_seq":2423,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823622+0000","last_change":"2024-03-19T03:00:04.517597+0000","last_active":"2024-03-19T03:06:09.823622+0000","last_peered":"2024-03-19T03:06:09.823622+0000","last_clean":"2024-03-19T03:06:09.823622+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T03:06:09.823622+0000","last_undegraded":"2024-03-19T03:06:09.823622+0000","last_fullsized":"2024-03-19T03:06:09.823622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:01:07.022928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053512995000000001,"stat_sum":{"num_bytes":2174562,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":27,"num_read_kb":19,"num_write":323,"num_write_kb":2244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"37'154","reported_seq":228,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.015888+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T03:05:15.015888+0000","last_peered":"2024-03-19T03:05:15.015888+0000","last_clean":"2024-03-19T03:05:15.015888+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T03:05:15.015888+0000","last_undegraded":"2024-03-19T03:05:15.015888+0000","last_fullsized":"2024-03-19T03:05:15.015888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":41000,"num_write":152,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"38'331","reported_seq":2443,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896906+0000","last_change":"2024-03-19T03:00:04.293854+0000","last_active":"2024-03-19T03:06:09.896906+0000","last_peered":"2024-03-19T03:06:09.896906+0000","last_clean":"2024-03-19T03:06:09.896906+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T03:06:09.896906+0000","last_undegraded":"2024-03-19T03:06:09.896906+0000","last_fullsized":"2024-03-19T03:06:09.896906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:32.540025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068156319000000007,"stat_sum":{"num_bytes":18375725,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":37,"num_read_kb":16403,"num_write":329,"num_write_kb":26314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"37'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.450142+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T03:05:10.450142+0000","last_peered":"2024-03-19T03:05:10.450142+0000","last_clean":"2024-03-19T03:05:10.450142+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T03:05:10.450142+0000","last_undegraded":"2024-03-19T03:05:10.450142+0000","last_fullsized":"2024-03-19T03:05:10.450142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"38'339","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819009+0000","last_change":"2024-03-19T03:00:04.518776+0000","last_active":"2024-03-19T03:06:09.819009+0000","last_peered":"2024-03-19T03:06:09.819009+0000","last_clean":"2024-03-19T03:06:09.819009+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T03:06:09.819009+0000","last_undegraded":"2024-03-19T03:06:09.819009+0000","last_fullsized":"2024-03-19T03:06:09.819009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.456673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051813761,"stat_sum":{"num_bytes":7359660,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":339,"num_write_kb":7364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"38'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.732969+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T03:05:09.732969+0000","last_peered":"2024-03-19T03:05:09.732969+0000","last_clean":"2024-03-19T03:05:09.732969+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T03:05:09.732969+0000","last_undegraded":"2024-03-19T03:05:09.732969+0000","last_fullsized":"2024-03-19T03:05:09.732969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"38'340","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.855709+0000","last_change":"2024-03-19T03:00:04.468344+0000","last_active":"2024-03-19T03:06:09.855709+0000","last_peered":"2024-03-19T03:06:09.855709+0000","last_clean":"2024-03-19T03:06:09.855709+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T03:06:09.855709+0000","last_undegraded":"2024-03-19T03:06:09.855709+0000","last_fullsized":"2024-03-19T03:06:09.855709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:34:09.179260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035899844,"stat_sum":{"num_bytes":22036853,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":37,"num_read_kb":12302,"num_write":335,"num_write_kb":21697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"38'128","reported_seq":215,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.376836+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T03:05:38.376836+0000","last_peered":"2024-03-19T03:05:38.376836+0000","last_clean":"2024-03-19T03:05:38.376836+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T03:05:38.376836+0000","last_undegraded":"2024-03-19T03:05:38.376836+0000","last_fullsized":"2024-03-19T03:05:38.376836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":31075,"num_write":133,"num_write_kb":8289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"38'357","reported_seq":2465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798019+0000","last_change":"2024-03-19T03:00:04.518831+0000","last_active":"2024-03-19T03:06:09.798019+0000","last_peered":"2024-03-19T03:06:09.798019+0000","last_clean":"2024-03-19T03:06:09.798019+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T03:06:09.798019+0000","last_undegraded":"2024-03-19T03:06:09.798019+0000","last_fullsized":"2024-03-19T03:06:09.798019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:39.124926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051911553999999999,"stat_sum":{"num_bytes":11598304,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":33,"num_read_kb":12305,"num_write":357,"num_write_kb":19699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"38'29","reported_seq":93,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.859256+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T03:05:51.859256+0000","last_peered":"2024-03-19T03:05:51.859256+0000","last_clean":"2024-03-19T03:05:51.859256+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T03:05:51.859256+0000","last_undegraded":"2024-03-19T03:05:51.859256+0000","last_fullsized":"2024-03-19T03:05:51.859256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":35,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"38'296","reported_seq":2390,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806192+0000","last_change":"2024-03-19T03:00:04.519069+0000","last_active":"2024-03-19T03:06:09.806192+0000","last_peered":"2024-03-19T03:06:09.806192+0000","last_clean":"2024-03-19T03:06:09.806192+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T03:06:09.806192+0000","last_undegraded":"2024-03-19T03:06:09.806192+0000","last_fullsized":"2024-03-19T03:06:09.806192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:27:49.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052296778000000002,"stat_sum":{"num_bytes":10391613,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":18,"num_read_kb":6155,"num_write":296,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"38'116","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:02.734657+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T03:06:02.734657+0000","last_peered":"2024-03-19T03:06:02.734657+0000","last_clean":"2024-03-19T03:06:02.734657+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T03:06:02.734657+0000","last_undegraded":"2024-03-19T03:06:02.734657+0000","last_fullsized":"2024-03-19T03:06:02.734657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":116,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"38'323","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.832152+0000","last_change":"2024-03-19T03:00:04.520259+0000","last_active":"2024-03-19T03:06:09.832152+0000","last_peered":"2024-03-19T03:06:09.832152+0000","last_clean":"2024-03-19T03:06:09.832152+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T03:06:09.832152+0000","last_undegraded":"2024-03-19T03:06:09.832152+0000","last_fullsized":"2024-03-19T03:06:09.832152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:46.091437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044895387000000002,"stat_sum":{"num_bytes":10001648,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":28,"num_read_kb":8211,"num_write":321,"num_write_kb":14027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"38'19","reported_seq":82,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856936+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T03:05:51.856936+0000","last_peered":"2024-03-19T03:05:51.856936+0000","last_clean":"2024-03-19T03:05:51.856936+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T03:05:51.856936+0000","last_undegraded":"2024-03-19T03:05:51.856936+0000","last_fullsized":"2024-03-19T03:05:51.856936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":20,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"38'362","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.803909+0000","last_change":"2024-03-19T03:00:04.462041+0000","last_active":"2024-03-19T03:06:09.803909+0000","last_peered":"2024-03-19T03:06:09.803909+0000","last_clean":"2024-03-19T03:06:09.803909+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T03:06:09.803909+0000","last_undegraded":"2024-03-19T03:06:09.803909+0000","last_fullsized":"2024-03-19T03:06:09.803909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:05:22.439007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055101615,"stat_sum":{"num_bytes":11152492,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":22,"num_read_kb":4110,"num_write":358,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"38'106","reported_seq":162,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.624147+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T03:06:01.624147+0000","last_peered":"2024-03-19T03:06:01.624147+0000","last_clean":"2024-03-19T03:06:01.624147+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T03:06:01.624147+0000","last_undegraded":"2024-03-19T03:06:01.624147+0000","last_fullsized":"2024-03-19T03:06:01.624147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":108,"num_write_kb":4216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'329","reported_seq":2435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828482+0000","last_change":"2024-03-19T03:00:04.517906+0000","last_active":"2024-03-19T03:06:09.828482+0000","last_peered":"2024-03-19T03:06:09.828482+0000","last_clean":"2024-03-19T03:06:09.828482+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T03:06:09.828482+0000","last_undegraded":"2024-03-19T03:06:09.828482+0000","last_fullsized":"2024-03-19T03:06:09.828482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:39.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051414362999999998,"stat_sum":{"num_bytes":15817335,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":30,"num_read_kb":4113,"num_write":329,"num_write_kb":19678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":211,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.447511+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T03:05:10.447511+0000","last_peered":"2024-03-19T03:05:10.447511+0000","last_clean":"2024-03-19T03:05:10.447511+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T03:05:10.447511+0000","last_undegraded":"2024-03-19T03:05:10.447511+0000","last_fullsized":"2024-03-19T03:05:10.447511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"38'1698","reported_seq":1756,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.922350+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T03:05:20.922350+0000","last_peered":"2024-03-19T03:05:20.922350+0000","last_clean":"2024-03-19T03:05:20.922350+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T03:05:20.922350+0000","last_undegraded":"2024-03-19T03:05:20.922350+0000","last_fullsized":"2024-03-19T03:05:20.922350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5356,"num_write":1694,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"38'324","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.913802+0000","last_change":"2024-03-19T03:00:04.637745+0000","last_active":"2024-03-19T03:06:09.913802+0000","last_peered":"2024-03-19T03:06:09.913802+0000","last_clean":"2024-03-19T03:06:09.913802+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T03:06:09.913802+0000","last_undegraded":"2024-03-19T03:06:09.913802+0000","last_fullsized":"2024-03-19T03:06:09.913802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:55:15.269339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063353052000000007,"stat_sum":{"num_bytes":11364310,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4110,"num_write":322,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'187","reported_seq":253,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.165979+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T03:05:15.165979+0000","last_peered":"2024-03-19T03:05:15.165979+0000","last_clean":"2024-03-19T03:05:15.165979+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T03:05:15.165979+0000","last_undegraded":"2024-03-19T03:05:15.165979+0000","last_fullsized":"2024-03-19T03:05:15.165979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":181,"num_write_kb":12402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"38'315","reported_seq":2411,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824543+0000","last_change":"2024-03-19T03:00:04.292884+0000","last_active":"2024-03-19T03:06:09.824543+0000","last_peered":"2024-03-19T03:06:09.824543+0000","last_clean":"2024-03-19T03:06:09.824543+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T03:06:09.824543+0000","last_undegraded":"2024-03-19T03:06:09.824543+0000","last_fullsized":"2024-03-19T03:06:09.824543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:15.770392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068696631999999994,"stat_sum":{"num_bytes":14176427,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":313,"num_write_kb":14010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"38'94","reported_seq":242,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.057114+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T03:05:20.057114+0000","last_peered":"2024-03-19T03:05:20.057114+0000","last_clean":"2024-03-19T03:05:20.057114+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T03:05:20.057114+0000","last_undegraded":"2024-03-19T03:05:20.057114+0000","last_fullsized":"2024-03-19T03:05:20.057114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":581,"num_write":90,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"38'324","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.785089+0000","last_change":"2024-03-19T03:00:04.732387+0000","last_active":"2024-03-19T03:06:09.785089+0000","last_peered":"2024-03-19T03:06:09.785089+0000","last_clean":"2024-03-19T03:06:09.785089+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T03:06:09.785089+0000","last_undegraded":"2024-03-19T03:06:09.785089+0000","last_fullsized":"2024-03-19T03:06:09.785089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:15.401411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044152405999999998,"stat_sum":{"num_bytes":9810356,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":324,"num_write_kb":9748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'61","reported_seq":125,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.331498+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T03:05:15.331498+0000","last_peered":"2024-03-19T03:05:15.331498+0000","last_clean":"2024-03-19T03:05:15.331498+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T03:05:15.331498+0000","last_undegraded":"2024-03-19T03:05:15.331498+0000","last_fullsized":"2024-03-19T03:05:15.331498+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":60,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"38'358","reported_seq":2462,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819761+0000","last_change":"2024-03-19T03:00:04.233217+0000","last_active":"2024-03-19T03:06:09.819761+0000","last_peered":"2024-03-19T03:06:09.819761+0000","last_clean":"2024-03-19T03:06:09.819761+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T03:06:09.819761+0000","last_undegraded":"2024-03-19T03:06:09.819761+0000","last_fullsized":"2024-03-19T03:06:09.819761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:01.512234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016334565999999998,"stat_sum":{"num_bytes":7144435,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":28,"num_read_kb":4112,"num_write":358,"num_write_kb":7164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"37'363","reported_seq":418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.705301+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T03:05:16.705301+0000","last_peered":"2024-03-19T03:05:16.705301+0000","last_clean":"2024-03-19T03:05:16.705301+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T03:05:16.705301+0000","last_undegraded":"2024-03-19T03:05:16.705301+0000","last_fullsized":"2024-03-19T03:05:16.705301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":360,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'337","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.796224+0000","last_change":"2024-03-19T03:00:04.461266+0000","last_active":"2024-03-19T03:06:09.796224+0000","last_peered":"2024-03-19T03:06:09.796224+0000","last_clean":"2024-03-19T03:06:09.796224+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T03:06:09.796224+0000","last_undegraded":"2024-03-19T03:06:09.796224+0000","last_fullsized":"2024-03-19T03:06:09.796224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:10:36.694537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054510613999999999,"stat_sum":{"num_bytes":18185428,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":28,"num_read_kb":9793,"num_write":337,"num_write_kb":22033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'364","reported_seq":424,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.763732+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T03:05:16.763732+0000","last_peered":"2024-03-19T03:05:16.763732+0000","last_clean":"2024-03-19T03:05:16.763732+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T03:05:16.763732+0000","last_undegraded":"2024-03-19T03:05:16.763732+0000","last_fullsized":"2024-03-19T03:05:16.763732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":360,"num_write_kb":8331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"38'347","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.775787+0000","last_change":"2024-03-19T03:00:04.730764+0000","last_active":"2024-03-19T03:06:09.775787+0000","last_peered":"2024-03-19T03:06:09.775787+0000","last_clean":"2024-03-19T03:06:09.775787+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T03:06:09.775787+0000","last_undegraded":"2024-03-19T03:06:09.775787+0000","last_fullsized":"2024-03-19T03:06:09.775787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:48.638029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047353098000000003,"stat_sum":{"num_bytes":7011110,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":27,"num_read_kb":4110,"num_write":348,"num_write_kb":6996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'268","reported_seq":336,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:05.909837+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T03:06:05.909837+0000","last_peered":"2024-03-19T03:06:05.909837+0000","last_clean":"2024-03-19T03:06:05.909837+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T03:06:05.909837+0000","last_undegraded":"2024-03-19T03:06:05.909837+0000","last_fullsized":"2024-03-19T03:06:05.909837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":638,"num_write":612,"num_write_kb":2118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"38'307","reported_seq":2400,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.779701+0000","last_change":"2024-03-19T03:00:04.266141+0000","last_active":"2024-03-19T03:06:09.779701+0000","last_peered":"2024-03-19T03:06:09.779701+0000","last_clean":"2024-03-19T03:06:09.779701+0000","last_became_active":"2024-03-19T02:55:45.402992+0000","last_became_peered":"2024-03-19T02:55:45.402992+0000","last_unstale":"2024-03-19T03:06:09.779701+0000","last_undegraded":"2024-03-19T03:06:09.779701+0000","last_fullsized":"2024-03-19T03:06:09.779701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:04.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050031514999999999,"stat_sum":{"num_bytes":18622985,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":24,"num_read_kb":16,"num_write":307,"num_write_kb":22442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'147","reported_seq":212,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.498499+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T03:05:40.498499+0000","last_peered":"2024-03-19T03:05:40.498499+0000","last_clean":"2024-03-19T03:05:40.498499+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T03:05:40.498499+0000","last_undegraded":"2024-03-19T03:05:40.498499+0000","last_fullsized":"2024-03-19T03:05:40.498499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":9370,"num_write":147,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"38'352","reported_seq":2467,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.888162+0000","last_change":"2024-03-19T03:00:04.295347+0000","last_active":"2024-03-19T03:06:09.888162+0000","last_peered":"2024-03-19T03:06:09.888162+0000","last_clean":"2024-03-19T03:06:09.888162+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.888162+0000","last_undegraded":"2024-03-19T03:06:09.888162+0000","last_fullsized":"2024-03-19T03:06:09.888162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:56:05.317379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068525034999999998,"stat_sum":{"num_bytes":35227376,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":40,"num_read_kb":16714,"num_write":347,"num_write_kb":34590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'17","reported_seq":76,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.056765+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T03:06:01.056765+0000","last_peered":"2024-03-19T03:06:01.056765+0000","last_clean":"2024-03-19T03:06:01.056765+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T03:06:01.056765+0000","last_undegraded":"2024-03-19T03:06:01.056765+0000","last_fullsized":"2024-03-19T03:06:01.056765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":20,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"38'345","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.788944+0000","last_change":"2024-03-19T03:00:04.233647+0000","last_active":"2024-03-19T03:06:09.788944+0000","last_peered":"2024-03-19T03:06:09.788944+0000","last_clean":"2024-03-19T03:06:09.788944+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T03:06:09.788944+0000","last_undegraded":"2024-03-19T03:06:09.788944+0000","last_fullsized":"2024-03-19T03:06:09.788944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:10.432157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016349961999999999,"stat_sum":{"num_bytes":23588150,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":30,"num_read_kb":20,"num_write":344,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"38'264","reported_seq":321,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856959+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T03:05:51.856959+0000","last_peered":"2024-03-19T03:05:51.856959+0000","last_clean":"2024-03-19T03:05:51.856959+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T03:05:51.856959+0000","last_undegraded":"2024-03-19T03:05:51.856959+0000","last_fullsized":"2024-03-19T03:05:51.856959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":264,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"38'344","reported_seq":2454,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.943424+0000","last_change":"2024-03-19T03:00:04.639224+0000","last_active":"2024-03-19T03:06:09.943424+0000","last_peered":"2024-03-19T03:06:09.943424+0000","last_clean":"2024-03-19T03:06:09.943424+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T03:06:09.943424+0000","last_undegraded":"2024-03-19T03:06:09.943424+0000","last_fullsized":"2024-03-19T03:06:09.943424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:17.069557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061753209000000003,"stat_sum":{"num_bytes":19421231,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":34,"num_read_kb":12304,"num_write":344,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"38'74","reported_seq":133,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.487245+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T03:05:38.487245+0000","last_peered":"2024-03-19T03:05:38.487245+0000","last_clean":"2024-03-19T03:05:38.487245+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T03:05:38.487245+0000","last_undegraded":"2024-03-19T03:05:38.487245+0000","last_fullsized":"2024-03-19T03:05:38.487245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":73,"num_write_kb":8285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"38'330","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.927473+0000","last_change":"2024-03-19T03:00:04.639280+0000","last_active":"2024-03-19T03:06:09.927473+0000","last_peered":"2024-03-19T03:06:09.927473+0000","last_clean":"2024-03-19T03:06:09.927473+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T03:06:09.927473+0000","last_undegraded":"2024-03-19T03:06:09.927473+0000","last_fullsized":"2024-03-19T03:06:09.927473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:22:15.157112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061740428,"stat_sum":{"num_bytes":14078096,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":23,"num_read_kb":8205,"num_write":330,"num_write_kb":13929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'281","reported_seq":351,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.024734+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T03:05:17.024734+0000","last_peered":"2024-03-19T03:05:17.024734+0000","last_clean":"2024-03-19T03:05:17.024734+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T03:05:17.024734+0000","last_undegraded":"2024-03-19T03:05:17.024734+0000","last_fullsized":"2024-03-19T03:05:17.024734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":40970,"num_write":274,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"38'342","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.719517+0000","last_change":"2024-03-19T03:00:04.467727+0000","last_active":"2024-03-19T03:06:09.719517+0000","last_peered":"2024-03-19T03:06:09.719517+0000","last_clean":"2024-03-19T03:06:09.719517+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T03:06:09.719517+0000","last_undegraded":"2024-03-19T03:06:09.719517+0000","last_fullsized":"2024-03-19T03:06:09.719517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:03.908237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035412037,"stat_sum":{"num_bytes":11182535,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":26,"num_read_kb":4109,"num_write":343,"num_write_kb":11053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"38'303","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824265+0000","last_change":"2024-03-19T03:00:04.460570+0000","last_active":"2024-03-19T03:06:09.824265+0000","last_peered":"2024-03-19T03:06:09.824265+0000","last_clean":"2024-03-19T03:06:09.824265+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T03:06:09.824265+0000","last_undegraded":"2024-03-19T03:06:09.824265+0000","last_fullsized":"2024-03-19T03:06:09.824265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:09:22.869183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053825845999999997,"stat_sum":{"num_bytes":7076024,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":303,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"37'88","reported_seq":152,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.432014+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T03:05:15.432014+0000","last_peered":"2024-03-19T03:05:15.432014+0000","last_clean":"2024-03-19T03:05:15.432014+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T03:05:15.432014+0000","last_undegraded":"2024-03-19T03:05:15.432014+0000","last_fullsized":"2024-03-19T03:05:15.432014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":92,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"38'342","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.772096+0000","last_change":"2024-03-19T03:00:04.460533+0000","last_active":"2024-03-19T03:06:09.772096+0000","last_peered":"2024-03-19T03:06:09.772096+0000","last_clean":"2024-03-19T03:06:09.772096+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T03:06:09.772096+0000","last_undegraded":"2024-03-19T03:06:09.772096+0000","last_fullsized":"2024-03-19T03:06:09.772096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:51:06.702309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053805551,"stat_sum":{"num_bytes":27334536,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":24,"num_read_kb":7741,"num_write":341,"num_write_kb":30967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"38'2315","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.196171+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T03:05:40.196171+0000","last_peered":"2024-03-19T03:05:40.196171+0000","last_clean":"2024-03-19T03:05:40.196171+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T03:05:40.196171+0000","last_undegraded":"2024-03-19T03:05:40.196171+0000","last_fullsized":"2024-03-19T03:05:40.196171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":49170,"num_write":2306,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"38'324","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.894896+0000","last_change":"2024-03-19T03:00:04.294754+0000","last_active":"2024-03-19T03:06:09.894896+0000","last_peered":"2024-03-19T03:06:09.894896+0000","last_clean":"2024-03-19T03:06:09.894896+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T03:06:09.894896+0000","last_undegraded":"2024-03-19T03:06:09.894896+0000","last_fullsized":"2024-03-19T03:06:09.894896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:06.724957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067674389000000001,"stat_sum":{"num_bytes":14508603,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":31,"num_read_kb":8211,"num_write":323,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"38'1551","reported_seq":1618,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.698465+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T03:06:01.698465+0000","last_peered":"2024-03-19T03:06:01.698465+0000","last_clean":"2024-03-19T03:06:01.698465+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T03:06:01.698465+0000","last_undegraded":"2024-03-19T03:06:01.698465+0000","last_fullsized":"2024-03-19T03:06:01.698465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":42561,"num_write":1549,"num_write_kb":13574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"38'331","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.899725+0000","last_change":"2024-03-19T03:00:04.636091+0000","last_active":"2024-03-19T03:06:09.899725+0000","last_peered":"2024-03-19T03:06:09.899725+0000","last_clean":"2024-03-19T03:06:09.899725+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T03:06:09.899725+0000","last_undegraded":"2024-03-19T03:06:09.899725+0000","last_fullsized":"2024-03-19T03:06:09.899725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:21:22.564399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065375344000000002,"stat_sum":{"num_bytes":15000004,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":21,"num_read_kb":8204,"num_write":332,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"37'148","reported_seq":225,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.050653+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T03:05:20.050653+0000","last_peered":"2024-03-19T03:05:20.050653+0000","last_clean":"2024-03-19T03:05:20.050653+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T03:05:20.050653+0000","last_undegraded":"2024-03-19T03:05:20.050653+0000","last_fullsized":"2024-03-19T03:05:20.050653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":45076,"num_write":147,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":939458329,"num_objects":10563,"num_object_clones":0,"num_object_copies":21126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10563,"num_whiteouts":0,"num_read":1809,"num_read_kb":363000,"num_write":21147,"num_write_kb":1062305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1709027328,"data_stored":1877384053,"data_compressed":164805636,"data_compressed_allocated":182276096,"data_compressed_original":405610496,"omap_allocated":0,"internal_metadata":0},"log_size":21178,"ondisk_log_size":21178,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":227028075,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":557,"num_read_kb":727830,"num_write":11856,"num_write_kb":232365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120127488,"data_stored":454056150,"data_compressed":53210578,"data_compressed_allocated":67698688,"data_compressed_original":401686528,"omap_allocated":0,"internal_metadata":0},"log_size":11549,"ondisk_log_size":11549,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332176,"kb_used_data":242696,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94039664,"statfs":{"total":96636764160,"available":96296615936,"internally_reserved":0,"allocated":248520704,"data_stored":332228426,"data_compressed":32896566,"data_compressed_allocated":38154240,"data_compressed_original":130297856,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283912,"kb_used_data":194496,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94087928,"statfs":{"total":96636764160,"available":96346038272,"internally_reserved":0,"allocated":199163904,"data_stored":260529012,"data_compressed":19584348,"data_compressed_allocated":23072768,"data_compressed_original":89657344,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364880,"kb_used_data":253192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94006960,"statfs":{"total":96636764160,"available":96263127040,"internally_reserved":0,"allocated":259268608,"data_stored":316361128,"data_compressed":23587682,"data_compressed_allocated":27353088,"data_compressed_original":92348416,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320448,"kb_used_data":253176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94051392,"statfs":{"total":96636764160,"available":96308625408,"internally_reserved":0,"allocated":259252224,"data_stored":332900836,"data_compressed":37042510,"data_compressed_allocated":41844736,"data_compressed_original":123842560,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336116,"kb_used_data":273004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94035724,"statfs":{"total":96636764160,"available":96292581376,"internally_reserved":0,"allocated":279556096,"data_stored":335606622,"data_compressed":26219015,"data_compressed_allocated":29872128,"data_compressed_original":93212672,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182320,"kb_used_data":137256,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94189520,"statfs":{"total":96636764160,"available":96450068480,"internally_reserved":0,"allocated":140550144,"data_stored":170537803,"data_compressed":15586883,"data_compressed_allocated":17731584,"data_compressed_original":51924992,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208528,"kb_used_data":163464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94163312,"statfs":{"total":96636764160,"available":96423231488,"internally_reserved":0,"allocated":167387136,"data_stored":233785712,"data_compressed":26814662,"data_compressed_allocated":30638080,"data_compressed_original":101830656,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326524,"kb_used_data":259252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94045316,"statfs":{"total":96636764160,"available":96302403584,"internally_reserved":0,"allocated":265474048,"data_stored":327658768,"data_compressed":31314452,"data_compressed_allocated":35168256,"data_compressed_original":104353792,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12947456,"data_stored":50332286,"data_compressed":6253471,"data_compressed_allocated":7770112,"data_compressed_original":45166592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":58720416,"data_compressed":7340199,"data_compressed_allocated":9289728,"data_compressed_original":54624256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":21505709,"data_compressed":2290904,"data_compressed_allocated":3022848,"data_compressed_original":17223680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11042816,"data_stored":46672205,"data_compressed":5800482,"data_compressed_allocated":7323648,"data_compressed_original":42962944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14008320,"data_stored":62914686,"data_compressed":7417000,"data_compressed_allocated":9461760,"data_compressed_original":58380288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":62914650,"data_compressed":7062565,"data_compressed_allocated":8916992,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20590592,"data_stored":71303840,"data_compressed":7958306,"data_compressed_allocated":10170368,"data_compressed_original":60891136,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":79692358,"data_compressed":9087651,"data_compressed_allocated":11743232,"data_compressed_original":71311360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256462848,"data_stored":282593685,"data_compressed":26461690,"data_compressed_allocated":29118464,"data_compressed_original":63418368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156610560,"data_stored":179655935,"data_compressed":20569111,"data_compressed_allocated":22691840,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135131136,"data_stored":152463714,"data_compressed":14071312,"data_compressed_allocated":15671296,"data_compressed_original":37679104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270573568,"data_stored":292209323,"data_compressed":21139415,"data_compressed_allocated":23465984,"data_compressed_original":52903936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245436416,"data_stored":273102592,"data_compressed":30129180,"data_compressed_allocated":33001472,"data_compressed_original":68767744,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238751744,"data_stored":256562920,"data_compressed":17028787,"data_compressed_allocated":19054592,"data_compressed_original":44527616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178155520,"data_stored":189041622,"data_compressed":11629622,"data_compressed_allocated":12906496,"data_compressed_original":28774400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":227905536,"data_stored":251754262,"data_compressed":23776519,"data_compressed_allocated":26365952,"data_compressed_original":58413056,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:06:13.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:06:14.067 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.066+0000 7f912da00640 1 Processor -- start 2024-03-19T03:06:14.068 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.066+0000 7f912da00640 1 -- start start 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f912da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9128154dc0 0x7f9128071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f912da00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91280c60d0 0x7f9128071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f912da00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9128072640 con 0x7f91280721b0 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f912da00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f912815b740 con 0x7f9128154dc0 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f912da00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f912815ba20 con 0x7f91280c60d0 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f911e000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91280c60d0 0x7f9128071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.069 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f9126a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9128154dc0 0x7f9128071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f911e000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f91280c60d0 0x7f9128071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39600/0 (socket says 172.21.15.92:39600) 2024-03-19T03:06:14.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.068+0000 7f911e000640 1 -- 172.21.15.92:0/926018374 learned_addr learned my addr 172.21.15.92:0/926018374 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.069+0000 7f911e000640 1 -- 172.21.15.92:0/926018374 >> v2:172.21.15.92:3300/0 conn(0x7f9128154dc0 msgr2=0x7f9128071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.069+0000 7f911e000640 1 --2- 172.21.15.92:0/926018374 >> v2:172.21.15.92:3300/0 conn(0x7f9128154dc0 0x7f9128071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.069+0000 7f911e000640 1 -- 172.21.15.92:0/926018374 >> v1:172.21.15.92:6789/0 conn(0x7f91280721b0 legacy=0x7f91280c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.069+0000 7f911e000640 1 -- 172.21.15.92:0/926018374 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91280c8400 con 0x7f91280c60d0 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.070+0000 7f911e000640 1 --2- 172.21.15.92:0/926018374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91280c60d0 0x7f9128071c70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9128075600 tx=0x7f911002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbc570f7cfe45d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.070+0000 7f9126000640 1 -- 172.21.15.92:0/926018374 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110035020 con 0x7f91280c60d0 2024-03-19T03:06:14.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.070+0000 7f9126000640 1 -- 172.21.15.92:0/926018374 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9110031870 con 0x7f91280c60d0 2024-03-19T03:06:14.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.070+0000 7f9126000640 1 -- 172.21.15.92:0/926018374 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110031c40 con 0x7f91280c60d0 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 -- 172.21.15.92:0/926018374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91280c60d0 msgr2=0x7f9128071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 --2- 172.21.15.92:0/926018374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91280c60d0 0x7f9128071c70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9128075600 tx=0x7f911002b910 comp rx=0 tx=0).stop 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 -- 172.21.15.92:0/926018374 shutdown_connections 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 --2- 172.21.15.92:0/926018374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91280c60d0 0x7f9128071c70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 --2- 172.21.15.92:0/926018374 >> v2:172.21.15.92:3300/0 conn(0x7f9128154dc0 0x7f9128071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 -- 172.21.15.92:0/926018374 >> 172.21.15.92:0/926018374 conn(0x7f91280c8d90 msgr2=0x7f91280c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 -- 172.21.15.92:0/926018374 shutdown_connections 2024-03-19T03:06:14.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.071+0000 7f912da00640 1 -- 172.21.15.92:0/926018374 wait complete. 2024-03-19T03:06:14.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.073+0000 7f912da00640 1 Processor -- start 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.073+0000 7f912da00640 1 -- start start 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91280c60d0 0x7f91281f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9128154dc0 0x7f9128147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91280c73a0 con 0x7f91280721b0 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f91281551a0 con 0x7f9128154dc0 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f912da00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f91280c7520 con 0x7f91280c60d0 2024-03-19T03:06:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33982/0 (socket says 172.21.15.92:33982) 2024-03-19T03:06:14.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 -- 172.21.15.92:0/143374966 learned_addr learned my addr 172.21.15.92:0/143374966 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:14.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9127400640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9128154dc0 0x7f9128147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f911e000640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91280c60d0 0x7f91281f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 -- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91280c60d0 msgr2=0x7f91281f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91280c60d0 0x7f91281f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 -- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9128154dc0 msgr2=0x7f9128147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9128154dc0 0x7f9128147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.074+0000 7f9126a00640 1 -- 172.21.15.92:0/143374966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91281537a0 con 0x7f91280721b0 2024-03-19T03:06:14.078 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f9126a00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f911800e870 tx=0x7f911800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118017070 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f911800c940 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91280cab40 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911800cd30 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.075+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91280cadb0 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.076+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91280684c0 con 0x7f91280721b0 2024-03-19T03:06:14.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.077+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f911801a070 con 0x7f91280721b0 2024-03-19T03:06:14.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.078+0000 7f911f400640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 0x7f910406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.078+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f911808aa60 con 0x7f91280721b0 2024-03-19T03:06:14.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.078+0000 7f911e000640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 0x7f910406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.079+0000 7f911e000640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 0x7f910406e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9110036000 tx=0x7f911003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.083+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9118058d30 con 0x7f91280721b0 2024-03-19T03:06:14.279 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.279+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f91280c66e0 con 0x7f910406bbd0 2024-03-19T03:06:14.287 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.286+0000 7f911f400640 1 -- 172.21.15.92:0/143374966 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256757 (secure 0 0 0) 0x7f9110086b10 con 0x7f910406bbd0 2024-03-19T03:06:14.290 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:14.292 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T03:06:14.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 msgr2=0x7f910406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.293 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 0x7f910406e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9110036000 tx=0x7f911003c040 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 msgr2=0x7f91280cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f911800e870 tx=0x7f911800ed30 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 shutdown_connections 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f910406bbd0 0x7f910406e090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9128154dc0 0x7f9128147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91280c60d0 0x7f91281f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 --2- 172.21.15.92:0/143374966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91280721b0 0x7f91280cb5b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.291+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 >> 172.21.15.92:0/143374966 conn(0x7f91280c8d90 msgr2=0x7f91280c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:14.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.292+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 shutdown_connections 2024-03-19T03:06:14.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.292+0000 7f912da00640 1 -- 172.21.15.92:0/143374966 wait complete. 2024-03-19T03:06:14.311 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-03-19T03:06:13.974360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1167076772,"num_objects":10705,"num_object_clones":0,"num_object_copies":21410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10705,"num_whiteouts":0,"num_read":2482,"num_read_kb":1090930,"num_write":33162,"num_write_kb":1297086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32844,"ondisk_log_size":32844,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2354904,"kb_used_data":1776536,"kb_used_omap":12,"kb_used_meta":578291,"kb_avail":752619816,"statfs":{"total":773094113280,"available":770682691584,"internally_reserved":0,"allocated":1819172864,"data_stored":2309608307,"data_compressed":213046118,"data_compressed_allocated":243834880,"data_compressed_original":787468288,"omap_allocated":12750,"internal_metadata":592170546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33868845,"num_objects":1890,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":34354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002256"},"pg_stats":[{"pgid":"3.3f","version":"38'340","reported_seq":2442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.831501+0000","last_change":"2024-03-19T03:00:04.520460+0000","last_active":"2024-03-19T03:06:09.831501+0000","last_peered":"2024-03-19T03:06:09.831501+0000","last_clean":"2024-03-19T03:06:09.831501+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T03:06:09.831501+0000","last_undegraded":"2024-03-19T03:06:09.831501+0000","last_fullsized":"2024-03-19T03:06:09.831501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:33.132110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045874986,"stat_sum":{"num_bytes":26339349,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":26,"num_read_kb":8207,"num_write":340,"num_write_kb":25903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"38'315","reported_seq":2418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.839204+0000","last_change":"2024-03-19T03:00:04.730981+0000","last_active":"2024-03-19T03:06:09.839204+0000","last_peered":"2024-03-19T03:06:09.839204+0000","last_clean":"2024-03-19T03:06:09.839204+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T03:06:09.839204+0000","last_undegraded":"2024-03-19T03:06:09.839204+0000","last_fullsized":"2024-03-19T03:06:09.839204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:40.160269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047342589999999997,"stat_sum":{"num_bytes":1430220,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":315,"num_write_kb":1566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"38'322","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.810784+0000","last_change":"2024-03-19T03:00:04.294476+0000","last_active":"2024-03-19T03:06:09.810784+0000","last_peered":"2024-03-19T03:06:09.810784+0000","last_clean":"2024-03-19T03:06:09.810784+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T03:06:09.810784+0000","last_undegraded":"2024-03-19T03:06:09.810784+0000","last_fullsized":"2024-03-19T03:06:09.810784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:54:31.834322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069861674999999998,"stat_sum":{"num_bytes":15481473,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":33,"num_read_kb":8211,"num_write":321,"num_write_kb":23482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"38'326","reported_seq":2421,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813773+0000","last_change":"2024-03-19T03:00:04.461050+0000","last_active":"2024-03-19T03:06:09.813773+0000","last_peered":"2024-03-19T03:06:09.813773+0000","last_clean":"2024-03-19T03:06:09.813773+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T03:06:09.813773+0000","last_undegraded":"2024-03-19T03:06:09.813773+0000","last_fullsized":"2024-03-19T03:06:09.813773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:49.707937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054309962000000003,"stat_sum":{"num_bytes":20068260,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":19,"num_read_kb":8201,"num_write":329,"num_write_kb":19659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"38'348","reported_seq":2448,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813079+0000","last_change":"2024-03-19T03:00:04.518539+0000","last_active":"2024-03-19T03:06:09.813079+0000","last_peered":"2024-03-19T03:06:09.813079+0000","last_clean":"2024-03-19T03:06:09.813079+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T03:06:09.813079+0000","last_undegraded":"2024-03-19T03:06:09.813079+0000","last_fullsized":"2024-03-19T03:06:09.813079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:46.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051753202999999998,"stat_sum":{"num_bytes":6591355,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":25,"num_read_kb":4114,"num_write":347,"num_write_kb":6624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"38'331","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823940+0000","last_change":"2024-03-19T03:00:04.458228+0000","last_active":"2024-03-19T03:06:09.823940+0000","last_peered":"2024-03-19T03:06:09.823940+0000","last_clean":"2024-03-19T03:06:09.823940+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T03:06:09.823940+0000","last_undegraded":"2024-03-19T03:06:09.823940+0000","last_fullsized":"2024-03-19T03:06:09.823940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:05:00.342939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051735362,"stat_sum":{"num_bytes":12769638,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":28,"num_read_kb":4115,"num_write":331,"num_write_kb":12643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"38'343","reported_seq":2440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.849648+0000","last_change":"2024-03-19T03:00:04.518609+0000","last_active":"2024-03-19T03:06:09.849648+0000","last_peered":"2024-03-19T03:06:09.849648+0000","last_clean":"2024-03-19T03:06:09.849648+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T03:06:09.849648+0000","last_undegraded":"2024-03-19T03:06:09.849648+0000","last_fullsized":"2024-03-19T03:06:09.849648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:41.608998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051913059999999997,"stat_sum":{"num_bytes":23638346,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":344,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"38'298","reported_seq":2388,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.795953+0000","last_change":"2024-03-19T03:00:04.268010+0000","last_active":"2024-03-19T03:06:09.795953+0000","last_peered":"2024-03-19T03:06:09.795953+0000","last_clean":"2024-03-19T03:06:09.795953+0000","last_became_active":"2024-03-19T02:55:45.392555+0000","last_became_peered":"2024-03-19T02:55:45.392555+0000","last_unstale":"2024-03-19T03:06:09.795953+0000","last_undegraded":"2024-03-19T03:06:09.795953+0000","last_fullsized":"2024-03-19T03:06:09.795953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:37.417889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052882827,"stat_sum":{"num_bytes":15124789,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":21,"num_read_kb":4106,"num_write":297,"num_write_kb":14929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"38'322","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828302+0000","last_change":"2024-03-19T03:00:04.520474+0000","last_active":"2024-03-19T03:06:09.828302+0000","last_peered":"2024-03-19T03:06:09.828302+0000","last_clean":"2024-03-19T03:06:09.828302+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T03:06:09.828302+0000","last_undegraded":"2024-03-19T03:06:09.828302+0000","last_fullsized":"2024-03-19T03:06:09.828302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:29.088176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045138699999999997,"stat_sum":{"num_bytes":8352128,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":30,"num_read_kb":4119,"num_write":322,"num_write_kb":12425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"38'304","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.757119+0000","last_change":"2024-03-19T03:00:04.469206+0000","last_active":"2024-03-19T03:06:09.757119+0000","last_peered":"2024-03-19T03:06:09.757119+0000","last_clean":"2024-03-19T03:06:09.757119+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T03:06:09.757119+0000","last_undegraded":"2024-03-19T03:06:09.757119+0000","last_fullsized":"2024-03-19T03:06:09.757119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.863780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033183580999999997,"stat_sum":{"num_bytes":17872088,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":8208,"num_write":302,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"38'305","reported_seq":2406,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.768960+0000","last_change":"2024-03-19T03:00:04.468803+0000","last_active":"2024-03-19T03:06:09.768960+0000","last_peered":"2024-03-19T03:06:09.768960+0000","last_clean":"2024-03-19T03:06:09.768960+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T03:06:09.768960+0000","last_undegraded":"2024-03-19T03:06:09.768960+0000","last_fullsized":"2024-03-19T03:06:09.768960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:05.106811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036038919000000003,"stat_sum":{"num_bytes":18621884,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":26,"num_read_kb":8210,"num_write":305,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"38'292","reported_seq":2385,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.761101+0000","last_change":"2024-03-19T03:00:04.233381+0000","last_active":"2024-03-19T03:06:09.761101+0000","last_peered":"2024-03-19T03:06:09.761101+0000","last_clean":"2024-03-19T03:06:09.761101+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T03:06:09.761101+0000","last_undegraded":"2024-03-19T03:06:09.761101+0000","last_fullsized":"2024-03-19T03:06:09.761101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:36:06.236162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016224016000000001,"stat_sum":{"num_bytes":10799086,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":18,"num_read_kb":10,"num_write":291,"num_write_kb":14795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"38'286","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.918837+0000","last_change":"2024-03-19T03:00:04.638962+0000","last_active":"2024-03-19T03:06:09.918837+0000","last_peered":"2024-03-19T03:06:09.918837+0000","last_clean":"2024-03-19T03:06:09.918837+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T03:06:09.918837+0000","last_undegraded":"2024-03-19T03:06:09.918837+0000","last_fullsized":"2024-03-19T03:06:09.918837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:12.950806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061626093999999999,"stat_sum":{"num_bytes":18869857,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":26,"num_read_kb":4115,"num_write":285,"num_write_kb":26775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"38'330","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.890035+0000","last_change":"2024-03-19T03:00:04.294731+0000","last_active":"2024-03-19T03:06:09.890035+0000","last_peered":"2024-03-19T03:06:09.890035+0000","last_clean":"2024-03-19T03:06:09.890035+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.890035+0000","last_undegraded":"2024-03-19T03:06:09.890035+0000","last_fullsized":"2024-03-19T03:06:09.890035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.987902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.06936341,"stat_sum":{"num_bytes":7021822,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":330,"num_write_kb":6980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"38'351","reported_seq":2453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.838055+0000","last_change":"2024-03-19T03:00:04.731871+0000","last_active":"2024-03-19T03:06:09.838055+0000","last_peered":"2024-03-19T03:06:09.838055+0000","last_clean":"2024-03-19T03:06:09.838055+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T03:06:09.838055+0000","last_undegraded":"2024-03-19T03:06:09.838055+0000","last_fullsized":"2024-03-19T03:06:09.838055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:26.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047989829999999997,"stat_sum":{"num_bytes":22976920,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":351,"num_write_kb":22623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"38'348","reported_seq":2450,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.868400+0000","last_change":"2024-03-19T03:00:04.463722+0000","last_active":"2024-03-19T03:06:09.868400+0000","last_peered":"2024-03-19T03:06:09.868400+0000","last_clean":"2024-03-19T03:06:09.868400+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T03:06:09.868400+0000","last_undegraded":"2024-03-19T03:06:09.868400+0000","last_fullsized":"2024-03-19T03:06:09.868400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:06.211614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054164370000000003,"stat_sum":{"num_bytes":10727184,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":27,"num_read_kb":4115,"num_write":346,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"38'344","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.771765+0000","last_change":"2024-03-19T03:00:04.463595+0000","last_active":"2024-03-19T03:06:09.771765+0000","last_peered":"2024-03-19T03:06:09.771765+0000","last_clean":"2024-03-19T03:06:09.771765+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T03:06:09.771765+0000","last_undegraded":"2024-03-19T03:06:09.771765+0000","last_fullsized":"2024-03-19T03:06:09.771765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:53:29.237872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054055032000000003,"stat_sum":{"num_bytes":10124226,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":32,"num_read_kb":4117,"num_write":343,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"38'290","reported_seq":2398,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798540+0000","last_change":"2024-03-19T03:00:04.265271+0000","last_active":"2024-03-19T03:06:09.798540+0000","last_peered":"2024-03-19T03:06:09.798540+0000","last_clean":"2024-03-19T03:06:09.798540+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T03:06:09.798540+0000","last_undegraded":"2024-03-19T03:06:09.798540+0000","last_fullsized":"2024-03-19T03:06:09.798540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:17.102471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.049230071,"stat_sum":{"num_bytes":7542726,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":289,"num_write_kb":7514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"38'322","reported_seq":2430,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828563+0000","last_change":"2024-03-19T03:00:04.518999+0000","last_active":"2024-03-19T03:06:09.828563+0000","last_peered":"2024-03-19T03:06:09.828563+0000","last_clean":"2024-03-19T03:06:09.828563+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T03:06:09.828563+0000","last_undegraded":"2024-03-19T03:06:09.828563+0000","last_fullsized":"2024-03-19T03:06:09.828563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:30.582720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050023194999999999,"stat_sum":{"num_bytes":14383098,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":32,"num_read_kb":16400,"num_write":321,"num_write_kb":18311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"38'352","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.764360+0000","last_change":"2024-03-19T03:00:04.469246+0000","last_active":"2024-03-19T03:06:09.764360+0000","last_peered":"2024-03-19T03:06:09.764360+0000","last_clean":"2024-03-19T03:06:09.764360+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T03:06:09.764360+0000","last_undegraded":"2024-03-19T03:06:09.764360+0000","last_fullsized":"2024-03-19T03:06:09.764360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:17:26.060817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033531875000000003,"stat_sum":{"num_bytes":23233989,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":33,"num_read_kb":12308,"num_write":352,"num_write_kb":31057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"38'318","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925503+0000","last_change":"2024-03-19T03:00:04.294471+0000","last_active":"2024-03-19T03:06:09.925503+0000","last_peered":"2024-03-19T03:06:09.925503+0000","last_clean":"2024-03-19T03:06:09.925503+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T03:06:09.925503+0000","last_undegraded":"2024-03-19T03:06:09.925503+0000","last_fullsized":"2024-03-19T03:06:09.925503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:35.963767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068442499000000004,"stat_sum":{"num_bytes":13700535,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":33,"num_read_kb":3602,"num_write":320,"num_write_kb":13502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"38'341","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806378+0000","last_change":"2024-03-19T03:00:04.460208+0000","last_active":"2024-03-19T03:06:09.806378+0000","last_peered":"2024-03-19T03:06:09.806378+0000","last_clean":"2024-03-19T03:06:09.806378+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T03:06:09.806378+0000","last_undegraded":"2024-03-19T03:06:09.806378+0000","last_fullsized":"2024-03-19T03:06:09.806378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:08.718707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053636608000000002,"stat_sum":{"num_bytes":9951503,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":340,"num_write_kb":13990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"38'325","reported_seq":2437,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.900508+0000","last_change":"2024-03-19T03:00:04.634734+0000","last_active":"2024-03-19T03:06:09.900508+0000","last_peered":"2024-03-19T03:06:09.900508+0000","last_clean":"2024-03-19T03:06:09.900508+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T03:06:09.900508+0000","last_undegraded":"2024-03-19T03:06:09.900508+0000","last_fullsized":"2024-03-19T03:06:09.900508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:56.724601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069022274999999994,"stat_sum":{"num_bytes":15469691,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":37,"num_read_kb":13386,"num_write":325,"num_write_kb":19374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"38'347","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.784715+0000","last_change":"2024-03-19T03:00:04.464949+0000","last_active":"2024-03-19T03:06:09.784715+0000","last_peered":"2024-03-19T03:06:09.784715+0000","last_clean":"2024-03-19T03:06:09.784715+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T03:06:09.784715+0000","last_undegraded":"2024-03-19T03:06:09.784715+0000","last_fullsized":"2024-03-19T03:06:09.784715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:40.894087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055361406000000002,"stat_sum":{"num_bytes":13836038,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":38,"num_read_kb":4325,"num_write":347,"num_write_kb":13690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"38'316","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896418+0000","last_change":"2024-03-19T03:00:04.636666+0000","last_active":"2024-03-19T03:06:09.896418+0000","last_peered":"2024-03-19T03:06:09.896418+0000","last_clean":"2024-03-19T03:06:09.896418+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T03:06:09.896418+0000","last_undegraded":"2024-03-19T03:06:09.896418+0000","last_fullsized":"2024-03-19T03:06:09.896418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:13:09.492010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066517528000000006,"stat_sum":{"num_bytes":14678298,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":314,"num_write_kb":14504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"38'357","reported_seq":2459,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.763988+0000","last_change":"2024-03-19T03:00:04.469451+0000","last_active":"2024-03-19T03:06:09.763988+0000","last_peered":"2024-03-19T03:06:09.763988+0000","last_clean":"2024-03-19T03:06:09.763988+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T03:06:09.763988+0000","last_undegraded":"2024-03-19T03:06:09.763988+0000","last_fullsized":"2024-03-19T03:06:09.763988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:08.415495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033380219000000003,"stat_sum":{"num_bytes":6267254,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":357,"num_write_kb":6310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"38'323","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.797829+0000","last_change":"2024-03-19T03:00:04.731007+0000","last_active":"2024-03-19T03:06:09.797829+0000","last_peered":"2024-03-19T03:06:09.797829+0000","last_clean":"2024-03-19T03:06:09.797829+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T03:06:09.797829+0000","last_undegraded":"2024-03-19T03:06:09.797829+0000","last_fullsized":"2024-03-19T03:06:09.797829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:38.782912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046987292,"stat_sum":{"num_bytes":20194837,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":27,"num_read_kb":4111,"num_write":325,"num_write_kb":23913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"38'312","reported_seq":2410,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925331+0000","last_change":"2024-03-19T03:00:04.639714+0000","last_active":"2024-03-19T03:06:09.925331+0000","last_peered":"2024-03-19T03:06:09.925331+0000","last_clean":"2024-03-19T03:06:09.925331+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T03:06:09.925331+0000","last_undegraded":"2024-03-19T03:06:09.925331+0000","last_fullsized":"2024-03-19T03:06:09.925331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:54.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054969302999999997,"stat_sum":{"num_bytes":19046740,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":22,"num_read_kb":7383,"num_write":312,"num_write_kb":22038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"38'348","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.926885+0000","last_change":"2024-03-19T03:00:04.295066+0000","last_active":"2024-03-19T03:06:09.926885+0000","last_peered":"2024-03-19T03:06:09.926885+0000","last_clean":"2024-03-19T03:06:09.926885+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T03:06:09.926885+0000","last_undegraded":"2024-03-19T03:06:09.926885+0000","last_fullsized":"2024-03-19T03:06:09.926885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:51.825764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068635079000000002,"stat_sum":{"num_bytes":22225330,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":36,"num_read_kb":12306,"num_write":348,"num_write_kb":21889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"38'345","reported_seq":2457,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781725+0000","last_change":"2024-03-19T03:00:04.460792+0000","last_active":"2024-03-19T03:06:09.781725+0000","last_peered":"2024-03-19T03:06:09.781725+0000","last_clean":"2024-03-19T03:06:09.781725+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T03:06:09.781725+0000","last_undegraded":"2024-03-19T03:06:09.781725+0000","last_fullsized":"2024-03-19T03:06:09.781725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:11.679527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054050365000000003,"stat_sum":{"num_bytes":13922214,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":36,"num_read_kb":4118,"num_write":345,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"38'394","reported_seq":2502,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.805290+0000","last_change":"2024-03-19T03:00:04.266403+0000","last_active":"2024-03-19T03:06:09.805290+0000","last_peered":"2024-03-19T03:06:09.805290+0000","last_clean":"2024-03-19T03:06:09.805290+0000","last_became_active":"2024-03-19T02:55:45.950634+0000","last_became_peered":"2024-03-19T02:55:45.950634+0000","last_unstale":"2024-03-19T03:06:09.805290+0000","last_undegraded":"2024-03-19T03:06:09.805290+0000","last_fullsized":"2024-03-19T03:06:09.805290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:59.262092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050892488,"stat_sum":{"num_bytes":11658804,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":27,"num_read_kb":4113,"num_write":395,"num_write_kb":15629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"38'360","reported_seq":2472,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.864349+0000","last_change":"2024-03-19T03:00:04.467477+0000","last_active":"2024-03-19T03:06:09.864349+0000","last_peered":"2024-03-19T03:06:09.864349+0000","last_clean":"2024-03-19T03:06:09.864349+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T03:06:09.864349+0000","last_undegraded":"2024-03-19T03:06:09.864349+0000","last_fullsized":"2024-03-19T03:06:09.864349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:19.601901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035464137999999999,"stat_sum":{"num_bytes":11414841,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":36,"num_read_kb":4123,"num_write":359,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"38'322","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781195+0000","last_change":"2024-03-19T03:00:04.233486+0000","last_active":"2024-03-19T03:06:09.781195+0000","last_peered":"2024-03-19T03:06:09.781195+0000","last_clean":"2024-03-19T03:06:09.781195+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T03:06:09.781195+0000","last_undegraded":"2024-03-19T03:06:09.781195+0000","last_fullsized":"2024-03-19T03:06:09.781195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.162762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016612378000000001,"stat_sum":{"num_bytes":15052974,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":23,"num_read_kb":4112,"num_write":322,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'49","reported_seq":112,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.737087+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T03:05:09.737087+0000","last_peered":"2024-03-19T03:05:09.737087+0000","last_clean":"2024-03-19T03:05:09.737087+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T03:05:09.737087+0000","last_undegraded":"2024-03-19T03:05:09.737087+0000","last_fullsized":"2024-03-19T03:05:09.737087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":52,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"38'306","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.922358+0000","last_change":"2024-03-19T03:00:04.295011+0000","last_active":"2024-03-19T03:06:09.922358+0000","last_peered":"2024-03-19T03:06:09.922358+0000","last_clean":"2024-03-19T03:06:09.922358+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T03:06:09.922358+0000","last_undegraded":"2024-03-19T03:06:09.922358+0000","last_fullsized":"2024-03-19T03:06:09.922358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:17.919923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066278250999999996,"stat_sum":{"num_bytes":6674098,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":15,"num_write":306,"num_write_kb":6677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"38'198","reported_seq":268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.188393+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T03:05:17.188393+0000","last_peered":"2024-03-19T03:05:17.188393+0000","last_clean":"2024-03-19T03:05:17.188393+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T03:05:17.188393+0000","last_undegraded":"2024-03-19T03:05:17.188393+0000","last_fullsized":"2024-03-19T03:05:17.188393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":61440,"num_write":198,"num_write_kb":16593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"38'29","reported_seq":87,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.904891+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T03:05:15.904891+0000","last_peered":"2024-03-19T03:05:15.904891+0000","last_clean":"2024-03-19T03:05:15.904891+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T03:05:15.904891+0000","last_undegraded":"2024-03-19T03:05:15.904891+0000","last_fullsized":"2024-03-19T03:05:15.904891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":29,"num_write_kb":4131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"38'386","reported_seq":2495,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.946964+0000","last_change":"2024-03-19T03:00:04.638741+0000","last_active":"2024-03-19T03:06:09.946964+0000","last_peered":"2024-03-19T03:06:09.946964+0000","last_clean":"2024-03-19T03:06:09.946964+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T03:06:09.946964+0000","last_undegraded":"2024-03-19T03:06:09.946964+0000","last_fullsized":"2024-03-19T03:06:09.946964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:31.894668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061005943,"stat_sum":{"num_bytes":22416031,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":34,"num_read_kb":16407,"num_write":386,"num_write_kb":31278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"38'751","reported_seq":814,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:06.635700+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T03:06:06.635700+0000","last_peered":"2024-03-19T03:06:06.635700+0000","last_clean":"2024-03-19T03:06:06.635700+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T03:06:06.635700+0000","last_undegraded":"2024-03-19T03:06:06.635700+0000","last_fullsized":"2024-03-19T03:06:06.635700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":748,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"38'321","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828476+0000","last_change":"2024-03-19T03:00:04.731670+0000","last_active":"2024-03-19T03:06:09.828476+0000","last_peered":"2024-03-19T03:06:09.828476+0000","last_clean":"2024-03-19T03:06:09.828476+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T03:06:09.828476+0000","last_undegraded":"2024-03-19T03:06:09.828476+0000","last_fullsized":"2024-03-19T03:06:09.828476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:37.837633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048289845999999997,"stat_sum":{"num_bytes":20433584,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":31,"num_read_kb":16402,"num_write":319,"num_write_kb":24181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'8","reported_seq":62,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.733040+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T03:05:09.733040+0000","last_peered":"2024-03-19T03:05:09.733040+0000","last_clean":"2024-03-19T03:05:09.733040+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T03:05:09.733040+0000","last_undegraded":"2024-03-19T03:05:09.733040+0000","last_fullsized":"2024-03-19T03:05:09.733040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"38'348","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.804516+0000","last_change":"2024-03-19T03:00:04.266848+0000","last_active":"2024-03-19T03:06:09.804516+0000","last_peered":"2024-03-19T03:06:09.804516+0000","last_clean":"2024-03-19T03:06:09.804516+0000","last_became_active":"2024-03-19T02:55:45.403591+0000","last_became_peered":"2024-03-19T02:55:45.403591+0000","last_unstale":"2024-03-19T03:06:09.804516+0000","last_undegraded":"2024-03-19T03:06:09.804516+0000","last_fullsized":"2024-03-19T03:06:09.804516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:03.822452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050955334999999997,"stat_sum":{"num_bytes":22779940,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":32,"num_read_kb":10328,"num_write":348,"num_write_kb":22426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"38'1621","reported_seq":1681,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.125122+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T03:05:54.125122+0000","last_peered":"2024-03-19T03:05:54.125122+0000","last_clean":"2024-03-19T03:05:54.125122+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T03:05:54.125122+0000","last_undegraded":"2024-03-19T03:05:54.125122+0000","last_fullsized":"2024-03-19T03:05:54.125122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":19565,"num_write":1617,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"38'308","reported_seq":2416,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.889708+0000","last_change":"2024-03-19T03:00:04.295350+0000","last_active":"2024-03-19T03:06:09.889708+0000","last_peered":"2024-03-19T03:06:09.889708+0000","last_clean":"2024-03-19T03:06:09.889708+0000","last_became_active":"2024-03-19T02:55:45.952615+0000","last_became_peered":"2024-03-19T02:55:45.952615+0000","last_unstale":"2024-03-19T03:06:09.889708+0000","last_undegraded":"2024-03-19T03:06:09.889708+0000","last_fullsized":"2024-03-19T03:06:09.889708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:35:42.703979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067870387000000004,"stat_sum":{"num_bytes":9678845,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":28,"num_read_kb":4114,"num_write":307,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"38'258","reported_seq":327,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:39.898558+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T03:05:39.898558+0000","last_peered":"2024-03-19T03:05:39.898558+0000","last_clean":"2024-03-19T03:05:39.898558+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T03:05:39.898558+0000","last_undegraded":"2024-03-19T03:05:39.898558+0000","last_fullsized":"2024-03-19T03:05:39.898558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":16407,"num_write":255,"num_write_kb":8344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"38'309","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.759258+0000","last_change":"2024-03-19T03:00:04.233584+0000","last_active":"2024-03-19T03:06:09.759258+0000","last_peered":"2024-03-19T03:06:09.759258+0000","last_clean":"2024-03-19T03:06:09.759258+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T03:06:09.759258+0000","last_undegraded":"2024-03-19T03:06:09.759258+0000","last_fullsized":"2024-03-19T03:06:09.759258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:28.958102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016376714000000001,"stat_sum":{"num_bytes":15448536,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":23,"num_read_kb":4110,"num_write":308,"num_write_kb":19342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"38'161","reported_seq":245,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.120316+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T03:05:54.120316+0000","last_peered":"2024-03-19T03:05:54.120316+0000","last_clean":"2024-03-19T03:05:54.120316+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T03:05:54.120316+0000","last_undegraded":"2024-03-19T03:05:54.120316+0000","last_fullsized":"2024-03-19T03:05:54.120316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":57372,"num_write":154,"num_write_kb":12387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"38'321","reported_seq":2423,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823622+0000","last_change":"2024-03-19T03:00:04.517597+0000","last_active":"2024-03-19T03:06:09.823622+0000","last_peered":"2024-03-19T03:06:09.823622+0000","last_clean":"2024-03-19T03:06:09.823622+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T03:06:09.823622+0000","last_undegraded":"2024-03-19T03:06:09.823622+0000","last_fullsized":"2024-03-19T03:06:09.823622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:01:07.022928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053512995000000001,"stat_sum":{"num_bytes":2174562,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":27,"num_read_kb":19,"num_write":323,"num_write_kb":2244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"37'154","reported_seq":228,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.015888+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T03:05:15.015888+0000","last_peered":"2024-03-19T03:05:15.015888+0000","last_clean":"2024-03-19T03:05:15.015888+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T03:05:15.015888+0000","last_undegraded":"2024-03-19T03:05:15.015888+0000","last_fullsized":"2024-03-19T03:05:15.015888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":41000,"num_write":152,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"38'331","reported_seq":2443,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896906+0000","last_change":"2024-03-19T03:00:04.293854+0000","last_active":"2024-03-19T03:06:09.896906+0000","last_peered":"2024-03-19T03:06:09.896906+0000","last_clean":"2024-03-19T03:06:09.896906+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T03:06:09.896906+0000","last_undegraded":"2024-03-19T03:06:09.896906+0000","last_fullsized":"2024-03-19T03:06:09.896906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:32.540025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068156319000000007,"stat_sum":{"num_bytes":18375725,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":37,"num_read_kb":16403,"num_write":329,"num_write_kb":26314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"37'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.450142+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T03:05:10.450142+0000","last_peered":"2024-03-19T03:05:10.450142+0000","last_clean":"2024-03-19T03:05:10.450142+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T03:05:10.450142+0000","last_undegraded":"2024-03-19T03:05:10.450142+0000","last_fullsized":"2024-03-19T03:05:10.450142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"38'339","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819009+0000","last_change":"2024-03-19T03:00:04.518776+0000","last_active":"2024-03-19T03:06:09.819009+0000","last_peered":"2024-03-19T03:06:09.819009+0000","last_clean":"2024-03-19T03:06:09.819009+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T03:06:09.819009+0000","last_undegraded":"2024-03-19T03:06:09.819009+0000","last_fullsized":"2024-03-19T03:06:09.819009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.456673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051813761,"stat_sum":{"num_bytes":7359660,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":339,"num_write_kb":7364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"38'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.732969+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T03:05:09.732969+0000","last_peered":"2024-03-19T03:05:09.732969+0000","last_clean":"2024-03-19T03:05:09.732969+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T03:05:09.732969+0000","last_undegraded":"2024-03-19T03:05:09.732969+0000","last_fullsized":"2024-03-19T03:05:09.732969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"38'340","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.855709+0000","last_change":"2024-03-19T03:00:04.468344+0000","last_active":"2024-03-19T03:06:09.855709+0000","last_peered":"2024-03-19T03:06:09.855709+0000","last_clean":"2024-03-19T03:06:09.855709+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T03:06:09.855709+0000","last_undegraded":"2024-03-19T03:06:09.855709+0000","last_fullsized":"2024-03-19T03:06:09.855709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:34:09.179260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035899844,"stat_sum":{"num_bytes":22036853,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":37,"num_read_kb":12302,"num_write":335,"num_write_kb":21697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"38'128","reported_seq":215,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.376836+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T03:05:38.376836+0000","last_peered":"2024-03-19T03:05:38.376836+0000","last_clean":"2024-03-19T03:05:38.376836+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T03:05:38.376836+0000","last_undegraded":"2024-03-19T03:05:38.376836+0000","last_fullsized":"2024-03-19T03:05:38.376836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":31075,"num_write":133,"num_write_kb":8289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"38'357","reported_seq":2465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798019+0000","last_change":"2024-03-19T03:00:04.518831+0000","last_active":"2024-03-19T03:06:09.798019+0000","last_peered":"2024-03-19T03:06:09.798019+0000","last_clean":"2024-03-19T03:06:09.798019+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T03:06:09.798019+0000","last_undegraded":"2024-03-19T03:06:09.798019+0000","last_fullsized":"2024-03-19T03:06:09.798019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:39.124926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051911553999999999,"stat_sum":{"num_bytes":11598304,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":33,"num_read_kb":12305,"num_write":357,"num_write_kb":19699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"38'29","reported_seq":93,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.859256+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T03:05:51.859256+0000","last_peered":"2024-03-19T03:05:51.859256+0000","last_clean":"2024-03-19T03:05:51.859256+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T03:05:51.859256+0000","last_undegraded":"2024-03-19T03:05:51.859256+0000","last_fullsized":"2024-03-19T03:05:51.859256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":35,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"38'296","reported_seq":2390,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806192+0000","last_change":"2024-03-19T03:00:04.519069+0000","last_active":"2024-03-19T03:06:09.806192+0000","last_peered":"2024-03-19T03:06:09.806192+0000","last_clean":"2024-03-19T03:06:09.806192+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T03:06:09.806192+0000","last_undegraded":"2024-03-19T03:06:09.806192+0000","last_fullsized":"2024-03-19T03:06:09.806192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:27:49.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052296778000000002,"stat_sum":{"num_bytes":10391613,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":18,"num_read_kb":6155,"num_write":296,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"38'116","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:02.734657+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T03:06:02.734657+0000","last_peered":"2024-03-19T03:06:02.734657+0000","last_clean":"2024-03-19T03:06:02.734657+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T03:06:02.734657+0000","last_undegraded":"2024-03-19T03:06:02.734657+0000","last_fullsized":"2024-03-19T03:06:02.734657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":116,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"38'323","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.832152+0000","last_change":"2024-03-19T03:00:04.520259+0000","last_active":"2024-03-19T03:06:09.832152+0000","last_peered":"2024-03-19T03:06:09.832152+0000","last_clean":"2024-03-19T03:06:09.832152+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T03:06:09.832152+0000","last_undegraded":"2024-03-19T03:06:09.832152+0000","last_fullsized":"2024-03-19T03:06:09.832152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:46.091437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044895387000000002,"stat_sum":{"num_bytes":10001648,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":28,"num_read_kb":8211,"num_write":321,"num_write_kb":14027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"38'19","reported_seq":82,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856936+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T03:05:51.856936+0000","last_peered":"2024-03-19T03:05:51.856936+0000","last_clean":"2024-03-19T03:05:51.856936+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T03:05:51.856936+0000","last_undegraded":"2024-03-19T03:05:51.856936+0000","last_fullsized":"2024-03-19T03:05:51.856936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":20,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"38'362","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.803909+0000","last_change":"2024-03-19T03:00:04.462041+0000","last_active":"2024-03-19T03:06:09.803909+0000","last_peered":"2024-03-19T03:06:09.803909+0000","last_clean":"2024-03-19T03:06:09.803909+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T03:06:09.803909+0000","last_undegraded":"2024-03-19T03:06:09.803909+0000","last_fullsized":"2024-03-19T03:06:09.803909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:05:22.439007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055101615,"stat_sum":{"num_bytes":11152492,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":22,"num_read_kb":4110,"num_write":358,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"38'106","reported_seq":162,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.624147+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T03:06:01.624147+0000","last_peered":"2024-03-19T03:06:01.624147+0000","last_clean":"2024-03-19T03:06:01.624147+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T03:06:01.624147+0000","last_undegraded":"2024-03-19T03:06:01.624147+0000","last_fullsized":"2024-03-19T03:06:01.624147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":108,"num_write_kb":4216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'329","reported_seq":2435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828482+0000","last_change":"2024-03-19T03:00:04.517906+0000","last_active":"2024-03-19T03:06:09.828482+0000","last_peered":"2024-03-19T03:06:09.828482+0000","last_clean":"2024-03-19T03:06:09.828482+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T03:06:09.828482+0000","last_undegraded":"2024-03-19T03:06:09.828482+0000","last_fullsized":"2024-03-19T03:06:09.828482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:39.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051414362999999998,"stat_sum":{"num_bytes":15817335,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":30,"num_read_kb":4113,"num_write":329,"num_write_kb":19678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":211,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.447511+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T03:05:10.447511+0000","last_peered":"2024-03-19T03:05:10.447511+0000","last_clean":"2024-03-19T03:05:10.447511+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T03:05:10.447511+0000","last_undegraded":"2024-03-19T03:05:10.447511+0000","last_fullsized":"2024-03-19T03:05:10.447511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"38'1698","reported_seq":1756,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.922350+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T03:05:20.922350+0000","last_peered":"2024-03-19T03:05:20.922350+0000","last_clean":"2024-03-19T03:05:20.922350+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T03:05:20.922350+0000","last_undegraded":"2024-03-19T03:05:20.922350+0000","last_fullsized":"2024-03-19T03:05:20.922350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5356,"num_write":1694,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"38'324","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.913802+0000","last_change":"2024-03-19T03:00:04.637745+0000","last_active":"2024-03-19T03:06:09.913802+0000","last_peered":"2024-03-19T03:06:09.913802+0000","last_clean":"2024-03-19T03:06:09.913802+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T03:06:09.913802+0000","last_undegraded":"2024-03-19T03:06:09.913802+0000","last_fullsized":"2024-03-19T03:06:09.913802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:55:15.269339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063353052000000007,"stat_sum":{"num_bytes":11364310,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4110,"num_write":322,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'187","reported_seq":253,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.165979+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T03:05:15.165979+0000","last_peered":"2024-03-19T03:05:15.165979+0000","last_clean":"2024-03-19T03:05:15.165979+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T03:05:15.165979+0000","last_undegraded":"2024-03-19T03:05:15.165979+0000","last_fullsized":"2024-03-19T03:05:15.165979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":181,"num_write_kb":12402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"38'315","reported_seq":2411,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824543+0000","last_change":"2024-03-19T03:00:04.292884+0000","last_active":"2024-03-19T03:06:09.824543+0000","last_peered":"2024-03-19T03:06:09.824543+0000","last_clean":"2024-03-19T03:06:09.824543+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T03:06:09.824543+0000","last_undegraded":"2024-03-19T03:06:09.824543+0000","last_fullsized":"2024-03-19T03:06:09.824543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:15.770392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068696631999999994,"stat_sum":{"num_bytes":14176427,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":313,"num_write_kb":14010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"38'94","reported_seq":242,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.057114+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T03:05:20.057114+0000","last_peered":"2024-03-19T03:05:20.057114+0000","last_clean":"2024-03-19T03:05:20.057114+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T03:05:20.057114+0000","last_undegraded":"2024-03-19T03:05:20.057114+0000","last_fullsized":"2024-03-19T03:05:20.057114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":581,"num_write":90,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"38'324","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.785089+0000","last_change":"2024-03-19T03:00:04.732387+0000","last_active":"2024-03-19T03:06:09.785089+0000","last_peered":"2024-03-19T03:06:09.785089+0000","last_clean":"2024-03-19T03:06:09.785089+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T03:06:09.785089+0000","last_undegraded":"2024-03-19T03:06:09.785089+0000","last_fullsized":"2024-03-19T03:06:09.785089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:15.401411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044152405999999998,"stat_sum":{"num_bytes":9810356,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":324,"num_write_kb":9748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'61","reported_seq":125,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.331498+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T03:05:15.331498+0000","last_peered":"2024-03-19T03:05:15.331498+0000","last_clean":"2024-03-19T03:05:15.331498+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T03:05:15.331498+0000","last_undegraded":"2024-03-19T03:05:15.331498+0000","last_fullsized":"2024-03-19T03:05:15.331498+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":60,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"38'358","reported_seq":2462,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819761+0000","last_change":"2024-03-19T03:00:04.233217+0000","last_active":"2024-03-19T03:06:09.819761+0000","last_peered":"2024-03-19T03:06:09.819761+0000","last_clean":"2024-03-19T03:06:09.819761+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T03:06:09.819761+0000","last_undegraded":"2024-03-19T03:06:09.819761+0000","last_fullsized":"2024-03-19T03:06:09.819761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:01.512234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016334565999999998,"stat_sum":{"num_bytes":7144435,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":28,"num_read_kb":4112,"num_write":358,"num_write_kb":7164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"37'363","reported_seq":418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.705301+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T03:05:16.705301+0000","last_peered":"2024-03-19T03:05:16.705301+0000","last_clean":"2024-03-19T03:05:16.705301+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T03:05:16.705301+0000","last_undegraded":"2024-03-19T03:05:16.705301+0000","last_fullsized":"2024-03-19T03:05:16.705301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":360,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'337","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.796224+0000","last_change":"2024-03-19T03:00:04.461266+0000","last_active":"2024-03-19T03:06:09.796224+0000","last_peered":"2024-03-19T03:06:09.796224+0000","last_clean":"2024-03-19T03:06:09.796224+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T03:06:09.796224+0000","last_undegraded":"2024-03-19T03:06:09.796224+0000","last_fullsized":"2024-03-19T03:06:09.796224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:10:36.694537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054510613999999999,"stat_sum":{"num_bytes":18185428,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":28,"num_read_kb":9793,"num_write":337,"num_write_kb":22033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'364","reported_seq":424,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.763732+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T03:05:16.763732+0000","last_peered":"2024-03-19T03:05:16.763732+0000","last_clean":"2024-03-19T03:05:16.763732+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T03:05:16.763732+0000","last_undegraded":"2024-03-19T03:05:16.763732+0000","last_fullsized":"2024-03-19T03:05:16.763732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":360,"num_write_kb":8331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"38'347","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.775787+0000","last_change":"2024-03-19T03:00:04.730764+0000","last_active":"2024-03-19T03:06:09.775787+0000","last_peered":"2024-03-19T03:06:09.775787+0000","last_clean":"2024-03-19T03:06:09.775787+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T03:06:09.775787+0000","last_undegraded":"2024-03-19T03:06:09.775787+0000","last_fullsized":"2024-03-19T03:06:09.775787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:48.638029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047353098000000003,"stat_sum":{"num_bytes":7011110,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":27,"num_read_kb":4110,"num_write":348,"num_write_kb":6996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'268","reported_seq":336,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:05.909837+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T03:06:05.909837+0000","last_peered":"2024-03-19T03:06:05.909837+0000","last_clean":"2024-03-19T03:06:05.909837+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T03:06:05.909837+0000","last_undegraded":"2024-03-19T03:06:05.909837+0000","last_fullsized":"2024-03-19T03:06:05.909837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":638,"num_write":612,"num_write_kb":2118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"38'307","reported_seq":2400,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.779701+0000","last_change":"2024-03-19T03:00:04.266141+0000","last_active":"2024-03-19T03:06:09.779701+0000","last_peered":"2024-03-19T03:06:09.779701+0000","last_clean":"2024-03-19T03:06:09.779701+0000","last_became_active":"2024-03-19T02:55:45.402992+0000","last_became_peered":"2024-03-19T02:55:45.402992+0000","last_unstale":"2024-03-19T03:06:09.779701+0000","last_undegraded":"2024-03-19T03:06:09.779701+0000","last_fullsized":"2024-03-19T03:06:09.779701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:04.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050031514999999999,"stat_sum":{"num_bytes":18622985,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":24,"num_read_kb":16,"num_write":307,"num_write_kb":22442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'147","reported_seq":212,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.498499+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T03:05:40.498499+0000","last_peered":"2024-03-19T03:05:40.498499+0000","last_clean":"2024-03-19T03:05:40.498499+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T03:05:40.498499+0000","last_undegraded":"2024-03-19T03:05:40.498499+0000","last_fullsized":"2024-03-19T03:05:40.498499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":9370,"num_write":147,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"38'352","reported_seq":2467,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.888162+0000","last_change":"2024-03-19T03:00:04.295347+0000","last_active":"2024-03-19T03:06:09.888162+0000","last_peered":"2024-03-19T03:06:09.888162+0000","last_clean":"2024-03-19T03:06:09.888162+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.888162+0000","last_undegraded":"2024-03-19T03:06:09.888162+0000","last_fullsized":"2024-03-19T03:06:09.888162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:56:05.317379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068525034999999998,"stat_sum":{"num_bytes":35227376,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":40,"num_read_kb":16714,"num_write":347,"num_write_kb":34590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'17","reported_seq":76,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.056765+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T03:06:01.056765+0000","last_peered":"2024-03-19T03:06:01.056765+0000","last_clean":"2024-03-19T03:06:01.056765+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T03:06:01.056765+0000","last_undegraded":"2024-03-19T03:06:01.056765+0000","last_fullsized":"2024-03-19T03:06:01.056765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":20,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"38'345","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.788944+0000","last_change":"2024-03-19T03:00:04.233647+0000","last_active":"2024-03-19T03:06:09.788944+0000","last_peered":"2024-03-19T03:06:09.788944+0000","last_clean":"2024-03-19T03:06:09.788944+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T03:06:09.788944+0000","last_undegraded":"2024-03-19T03:06:09.788944+0000","last_fullsized":"2024-03-19T03:06:09.788944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:10.432157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016349961999999999,"stat_sum":{"num_bytes":23588150,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":30,"num_read_kb":20,"num_write":344,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"38'264","reported_seq":321,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856959+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T03:05:51.856959+0000","last_peered":"2024-03-19T03:05:51.856959+0000","last_clean":"2024-03-19T03:05:51.856959+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T03:05:51.856959+0000","last_undegraded":"2024-03-19T03:05:51.856959+0000","last_fullsized":"2024-03-19T03:05:51.856959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":264,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"38'344","reported_seq":2454,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.943424+0000","last_change":"2024-03-19T03:00:04.639224+0000","last_active":"2024-03-19T03:06:09.943424+0000","last_peered":"2024-03-19T03:06:09.943424+0000","last_clean":"2024-03-19T03:06:09.943424+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T03:06:09.943424+0000","last_undegraded":"2024-03-19T03:06:09.943424+0000","last_fullsized":"2024-03-19T03:06:09.943424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:17.069557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061753209000000003,"stat_sum":{"num_bytes":19421231,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":34,"num_read_kb":12304,"num_write":344,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"38'74","reported_seq":133,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.487245+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T03:05:38.487245+0000","last_peered":"2024-03-19T03:05:38.487245+0000","last_clean":"2024-03-19T03:05:38.487245+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T03:05:38.487245+0000","last_undegraded":"2024-03-19T03:05:38.487245+0000","last_fullsized":"2024-03-19T03:05:38.487245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":73,"num_write_kb":8285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"38'330","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.927473+0000","last_change":"2024-03-19T03:00:04.639280+0000","last_active":"2024-03-19T03:06:09.927473+0000","last_peered":"2024-03-19T03:06:09.927473+0000","last_clean":"2024-03-19T03:06:09.927473+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T03:06:09.927473+0000","last_undegraded":"2024-03-19T03:06:09.927473+0000","last_fullsized":"2024-03-19T03:06:09.927473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:22:15.157112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061740428,"stat_sum":{"num_bytes":14078096,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":23,"num_read_kb":8205,"num_write":330,"num_write_kb":13929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'281","reported_seq":351,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.024734+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T03:05:17.024734+0000","last_peered":"2024-03-19T03:05:17.024734+0000","last_clean":"2024-03-19T03:05:17.024734+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T03:05:17.024734+0000","last_undegraded":"2024-03-19T03:05:17.024734+0000","last_fullsized":"2024-03-19T03:05:17.024734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":40970,"num_write":274,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"38'342","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.719517+0000","last_change":"2024-03-19T03:00:04.467727+0000","last_active":"2024-03-19T03:06:09.719517+0000","last_peered":"2024-03-19T03:06:09.719517+0000","last_clean":"2024-03-19T03:06:09.719517+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T03:06:09.719517+0000","last_undegraded":"2024-03-19T03:06:09.719517+0000","last_fullsized":"2024-03-19T03:06:09.719517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:03.908237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035412037,"stat_sum":{"num_bytes":11182535,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":26,"num_read_kb":4109,"num_write":343,"num_write_kb":11053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"38'303","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824265+0000","last_change":"2024-03-19T03:00:04.460570+0000","last_active":"2024-03-19T03:06:09.824265+0000","last_peered":"2024-03-19T03:06:09.824265+0000","last_clean":"2024-03-19T03:06:09.824265+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T03:06:09.824265+0000","last_undegraded":"2024-03-19T03:06:09.824265+0000","last_fullsized":"2024-03-19T03:06:09.824265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:09:22.869183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053825845999999997,"stat_sum":{"num_bytes":7076024,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":303,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"37'88","reported_seq":152,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.432014+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T03:05:15.432014+0000","last_peered":"2024-03-19T03:05:15.432014+0000","last_clean":"2024-03-19T03:05:15.432014+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T03:05:15.432014+0000","last_undegraded":"2024-03-19T03:05:15.432014+0000","last_fullsized":"2024-03-19T03:05:15.432014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":92,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"38'342","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.772096+0000","last_change":"2024-03-19T03:00:04.460533+0000","last_active":"2024-03-19T03:06:09.772096+0000","last_peered":"2024-03-19T03:06:09.772096+0000","last_clean":"2024-03-19T03:06:09.772096+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T03:06:09.772096+0000","last_undegraded":"2024-03-19T03:06:09.772096+0000","last_fullsized":"2024-03-19T03:06:09.772096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:51:06.702309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053805551,"stat_sum":{"num_bytes":27334536,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":24,"num_read_kb":7741,"num_write":341,"num_write_kb":30967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"38'2315","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.196171+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T03:05:40.196171+0000","last_peered":"2024-03-19T03:05:40.196171+0000","last_clean":"2024-03-19T03:05:40.196171+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T03:05:40.196171+0000","last_undegraded":"2024-03-19T03:05:40.196171+0000","last_fullsized":"2024-03-19T03:05:40.196171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":49170,"num_write":2306,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"38'324","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.894896+0000","last_change":"2024-03-19T03:00:04.294754+0000","last_active":"2024-03-19T03:06:09.894896+0000","last_peered":"2024-03-19T03:06:09.894896+0000","last_clean":"2024-03-19T03:06:09.894896+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T03:06:09.894896+0000","last_undegraded":"2024-03-19T03:06:09.894896+0000","last_fullsized":"2024-03-19T03:06:09.894896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:06.724957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067674389000000001,"stat_sum":{"num_bytes":14508603,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":31,"num_read_kb":8211,"num_write":323,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"38'1551","reported_seq":1618,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.698465+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T03:06:01.698465+0000","last_peered":"2024-03-19T03:06:01.698465+0000","last_clean":"2024-03-19T03:06:01.698465+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T03:06:01.698465+0000","last_undegraded":"2024-03-19T03:06:01.698465+0000","last_fullsized":"2024-03-19T03:06:01.698465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":42561,"num_write":1549,"num_write_kb":13574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"38'331","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.899725+0000","last_change":"2024-03-19T03:00:04.636091+0000","last_active":"2024-03-19T03:06:09.899725+0000","last_peered":"2024-03-19T03:06:09.899725+0000","last_clean":"2024-03-19T03:06:09.899725+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T03:06:09.899725+0000","last_undegraded":"2024-03-19T03:06:09.899725+0000","last_fullsized":"2024-03-19T03:06:09.899725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:21:22.564399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065375344000000002,"stat_sum":{"num_bytes":15000004,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":21,"num_read_kb":8204,"num_write":332,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"37'148","reported_seq":225,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.050653+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T03:05:20.050653+0000","last_peered":"2024-03-19T03:05:20.050653+0000","last_clean":"2024-03-19T03:05:20.050653+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T03:05:20.050653+0000","last_undegraded":"2024-03-19T03:05:20.050653+0000","last_fullsized":"2024-03-19T03:05:20.050653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":45076,"num_write":147,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":939458329,"num_objects":10563,"num_object_clones":0,"num_object_copies":21126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10563,"num_whiteouts":0,"num_read":1809,"num_read_kb":363000,"num_write":21147,"num_write_kb":1062305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1709027328,"data_stored":1877384053,"data_compressed":164805636,"data_compressed_allocated":182276096,"data_compressed_original":405610496,"omap_allocated":0,"internal_metadata":0},"log_size":21178,"ondisk_log_size":21178,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":227028075,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":557,"num_read_kb":727830,"num_write":11856,"num_write_kb":232365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120127488,"data_stored":454056150,"data_compressed":53210578,"data_compressed_allocated":67698688,"data_compressed_original":401686528,"omap_allocated":0,"internal_metadata":0},"log_size":11549,"ondisk_log_size":11549,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332176,"kb_used_data":242696,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94039664,"statfs":{"total":96636764160,"available":96296615936,"internally_reserved":0,"allocated":248520704,"data_stored":332228426,"data_compressed":32896566,"data_compressed_allocated":38154240,"data_compressed_original":130297856,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283912,"kb_used_data":194496,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94087928,"statfs":{"total":96636764160,"available":96346038272,"internally_reserved":0,"allocated":199163904,"data_stored":260529012,"data_compressed":19584348,"data_compressed_allocated":23072768,"data_compressed_original":89657344,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364880,"kb_used_data":253192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94006960,"statfs":{"total":96636764160,"available":96263127040,"internally_reserved":0,"allocated":259268608,"data_stored":316361128,"data_compressed":23587682,"data_compressed_allocated":27353088,"data_compressed_original":92348416,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320448,"kb_used_data":253176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94051392,"statfs":{"total":96636764160,"available":96308625408,"internally_reserved":0,"allocated":259252224,"data_stored":332900836,"data_compressed":37042510,"data_compressed_allocated":41844736,"data_compressed_original":123842560,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336116,"kb_used_data":273004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94035724,"statfs":{"total":96636764160,"available":96292581376,"internally_reserved":0,"allocated":279556096,"data_stored":335606622,"data_compressed":26219015,"data_compressed_allocated":29872128,"data_compressed_original":93212672,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182320,"kb_used_data":137256,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94189520,"statfs":{"total":96636764160,"available":96450068480,"internally_reserved":0,"allocated":140550144,"data_stored":170537803,"data_compressed":15586883,"data_compressed_allocated":17731584,"data_compressed_original":51924992,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208528,"kb_used_data":163464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94163312,"statfs":{"total":96636764160,"available":96423231488,"internally_reserved":0,"allocated":167387136,"data_stored":233785712,"data_compressed":26814662,"data_compressed_allocated":30638080,"data_compressed_original":101830656,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326524,"kb_used_data":259252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94045316,"statfs":{"total":96636764160,"available":96302403584,"internally_reserved":0,"allocated":265474048,"data_stored":327658768,"data_compressed":31314452,"data_compressed_allocated":35168256,"data_compressed_original":104353792,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12947456,"data_stored":50332286,"data_compressed":6253471,"data_compressed_allocated":7770112,"data_compressed_original":45166592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":58720416,"data_compressed":7340199,"data_compressed_allocated":9289728,"data_compressed_original":54624256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":21505709,"data_compressed":2290904,"data_compressed_allocated":3022848,"data_compressed_original":17223680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11042816,"data_stored":46672205,"data_compressed":5800482,"data_compressed_allocated":7323648,"data_compressed_original":42962944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14008320,"data_stored":62914686,"data_compressed":7417000,"data_compressed_allocated":9461760,"data_compressed_original":58380288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":62914650,"data_compressed":7062565,"data_compressed_allocated":8916992,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20590592,"data_stored":71303840,"data_compressed":7958306,"data_compressed_allocated":10170368,"data_compressed_original":60891136,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":79692358,"data_compressed":9087651,"data_compressed_allocated":11743232,"data_compressed_original":71311360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256462848,"data_stored":282593685,"data_compressed":26461690,"data_compressed_allocated":29118464,"data_compressed_original":63418368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156610560,"data_stored":179655935,"data_compressed":20569111,"data_compressed_allocated":22691840,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135131136,"data_stored":152463714,"data_compressed":14071312,"data_compressed_allocated":15671296,"data_compressed_original":37679104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270573568,"data_stored":292209323,"data_compressed":21139415,"data_compressed_allocated":23465984,"data_compressed_original":52903936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245436416,"data_stored":273102592,"data_compressed":30129180,"data_compressed_allocated":33001472,"data_compressed_original":68767744,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238751744,"data_stored":256562920,"data_compressed":17028787,"data_compressed_allocated":19054592,"data_compressed_original":44527616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178155520,"data_stored":189041622,"data_compressed":11629622,"data_compressed_allocated":12906496,"data_compressed_original":28774400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":227905536,"data_stored":251754262,"data_compressed":23776519,"data_compressed_allocated":26365952,"data_compressed_original":58413056,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:06:14.315 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T03:06:14.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:06:14.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.563+0000 7fda80600640 1 Processor -- start 2024-03-19T03:06:14.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.564+0000 7fda80600640 1 -- start start 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda80600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fda780c6470 0x7fda780c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda80600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda780c9a80 con 0x7fda781593d0 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda80600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda780c8650 con 0x7fda78152bf0 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda80600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fda780c6c90 con 0x7fda780c6470 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda7dc00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fda78152bf0 0x7fda7806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54498/0 (socket says 172.21.15.92:54498) 2024-03-19T03:06:14.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda7dc00640 1 -- 172.21.15.92:0/3708272473 learned_addr learned my addr 172.21.15.92:0/3708272473 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.565+0000 7fda7d200640 1 --2- 172.21.15.92:0/3708272473 >> v2:172.21.15.92:3300/0 conn(0x7fda780c6470 0x7fda780c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.566+0000 7fda77e00640 1 -- 172.21.15.92:0/3708272473 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101387945 0 0) 0x7fda780c8650 con 0x7fda78152bf0 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.566+0000 7fda77e00640 1 -- 172.21.15.92:0/3708272473 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda5c003520 con 0x7fda78152bf0 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.566+0000 7fda7d200640 1 -- 172.21.15.92:0/3708272473 >> v1:172.21.15.159:6789/0 conn(0x7fda78152bf0 legacy=0x7fda7806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.566+0000 7fda7d200640 1 -- 172.21.15.92:0/3708272473 >> v1:172.21.15.92:6789/0 conn(0x7fda781593d0 legacy=0x7fda780c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.566+0000 7fda7d200640 1 -- 172.21.15.92:0/3708272473 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda780c8930 con 0x7fda780c6470 2024-03-19T03:06:14.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.567+0000 7fda7d200640 1 --2- 172.21.15.92:0/3708272473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda780c6470 0x7fda780c3f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fda640097a0 tx=0x7fda6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558f8a8faf671bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.569 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.567+0000 7fda77e00640 1 -- 172.21.15.92:0/3708272473 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64035020 con 0x7fda780c6470 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.567+0000 7fda77e00640 1 -- 172.21.15.92:0/3708272473 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda64031830 con 0x7fda780c6470 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.567+0000 7fda77e00640 1 -- 172.21.15.92:0/3708272473 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64031c00 con 0x7fda780c6470 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 -- 172.21.15.92:0/3708272473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda780c6470 msgr2=0x7fda780c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 --2- 172.21.15.92:0/3708272473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda780c6470 0x7fda780c3f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fda640097a0 tx=0x7fda6402b8d0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 -- 172.21.15.92:0/3708272473 shutdown_connections 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 --2- 172.21.15.92:0/3708272473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda780c6470 0x7fda780c3f80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 -- 172.21.15.92:0/3708272473 >> 172.21.15.92:0/3708272473 conn(0x7fda780c8d90 msgr2=0x7fda780c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:14.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 -- 172.21.15.92:0/3708272473 shutdown_connections 2024-03-19T03:06:14.571 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.568+0000 7fda80600640 1 -- 172.21.15.92:0/3708272473 wait complete. 2024-03-19T03:06:14.572 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.570+0000 7fda80600640 1 Processor -- start 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.570+0000 7fda80600640 1 -- start start 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda78152bf0 0x7fda781427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda781593d0 0x7fda781f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda7806e4a0 con 0x7fda781593d0 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fda78159910 con 0x7fda780c6470 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda80600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fda7806ea60 con 0x7fda78152bf0 2024-03-19T03:06:14.573 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda781593d0 0x7fda781f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7d200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda78152bf0 0x7fda781427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39612/0 (socket says 172.21.15.92:39612) 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 -- 172.21.15.92:0/3705947679 learned_addr learned my addr 172.21.15.92:0/3705947679 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 -- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda78152bf0 msgr2=0x7fda781427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.574 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda78152bf0 0x7fda781427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 -- 172.21.15.92:0/3705947679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda781593d0 msgr2=0x7fda781f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda781593d0 0x7fda781f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.571+0000 7fda7dc00640 1 -- 172.21.15.92:0/3705947679 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda780cb270 con 0x7fda780c6470 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda7dc00640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda6800c500 tx=0x7fda6800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68017070 con 0x7fda780c6470 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda68005ce0 con 0x7fda780c6470 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda780cc070 con 0x7fda780c6470 2024-03-19T03:06:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68007330 con 0x7fda780c6470 2024-03-19T03:06:14.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.572+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda780cb7d0 con 0x7fda780c6470 2024-03-19T03:06:14.576 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.573+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda780684c0 con 0x7fda780c6470 2024-03-19T03:06:14.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.574+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda6801a070 con 0x7fda780c6470 2024-03-19T03:06:14.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.575+0000 7fda75600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 0x7fda5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:14.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.575+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fda6808a090 con 0x7fda780c6470 2024-03-19T03:06:14.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.575+0000 7fda7d200640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 0x7fda5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:14.577 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.576+0000 7fda7d200640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 0x7fda5806e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda64036000 tx=0x7fda64008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:14.583 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.583+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda780cb270 con 0x7fda780c6470 2024-03-19T03:06:14.778 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.778+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda78150ec0 con 0x7fda5806bbd0 2024-03-19T03:06:14.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.785+0000 7fda75600640 1 -- 172.21.15.92:0/3705947679 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256757 (secure 0 0 0) 0x7fda78150ec0 con 0x7fda5806bbd0 2024-03-19T03:06:14.790 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:14.791 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T03:06:14.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 msgr2=0x7fda5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 0x7fda5806e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda64036000 tx=0x7fda64008030 comp rx=0 tx=0).stop 2024-03-19T03:06:14.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 msgr2=0x7fda781578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda6800c500 tx=0x7fda6800c9c0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 shutdown_connections 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fda5806bbd0 0x7fda5806e090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda781593d0 0x7fda781f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fda78152bf0 0x7fda781427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 --2- 172.21.15.92:0/3705947679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fda780c6470 0x7fda781578c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:14.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 >> 172.21.15.92:0/3705947679 conn(0x7fda780c8d90 msgr2=0x7fda780c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:14.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 shutdown_connections 2024-03-19T03:06:14.795 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:14.792+0000 7fda80600640 1 -- 172.21.15.92:0/3705947679 wait complete. 2024-03-19T03:06:14.811 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-03-19T03:06:13.974360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1167076772,"num_objects":10705,"num_object_clones":0,"num_object_copies":21410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10705,"num_whiteouts":0,"num_read":2482,"num_read_kb":1090930,"num_write":33162,"num_write_kb":1297086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32844,"ondisk_log_size":32844,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2354904,"kb_used_data":1776536,"kb_used_omap":12,"kb_used_meta":578291,"kb_avail":752619816,"statfs":{"total":773094113280,"available":770682691584,"internally_reserved":0,"allocated":1819172864,"data_stored":2309608307,"data_compressed":213046118,"data_compressed_allocated":243834880,"data_compressed_original":787468288,"omap_allocated":12750,"internal_metadata":592170546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33868845,"num_objects":1890,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":34354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002256"},"pg_stats":[{"pgid":"3.3f","version":"38'340","reported_seq":2442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.831501+0000","last_change":"2024-03-19T03:00:04.520460+0000","last_active":"2024-03-19T03:06:09.831501+0000","last_peered":"2024-03-19T03:06:09.831501+0000","last_clean":"2024-03-19T03:06:09.831501+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T03:06:09.831501+0000","last_undegraded":"2024-03-19T03:06:09.831501+0000","last_fullsized":"2024-03-19T03:06:09.831501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:33.132110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045874986,"stat_sum":{"num_bytes":26339349,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":26,"num_read_kb":8207,"num_write":340,"num_write_kb":25903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"38'315","reported_seq":2418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.839204+0000","last_change":"2024-03-19T03:00:04.730981+0000","last_active":"2024-03-19T03:06:09.839204+0000","last_peered":"2024-03-19T03:06:09.839204+0000","last_clean":"2024-03-19T03:06:09.839204+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T03:06:09.839204+0000","last_undegraded":"2024-03-19T03:06:09.839204+0000","last_fullsized":"2024-03-19T03:06:09.839204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:40.160269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047342589999999997,"stat_sum":{"num_bytes":1430220,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":315,"num_write_kb":1566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"38'322","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.810784+0000","last_change":"2024-03-19T03:00:04.294476+0000","last_active":"2024-03-19T03:06:09.810784+0000","last_peered":"2024-03-19T03:06:09.810784+0000","last_clean":"2024-03-19T03:06:09.810784+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T03:06:09.810784+0000","last_undegraded":"2024-03-19T03:06:09.810784+0000","last_fullsized":"2024-03-19T03:06:09.810784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:54:31.834322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069861674999999998,"stat_sum":{"num_bytes":15481473,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":33,"num_read_kb":8211,"num_write":321,"num_write_kb":23482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"38'326","reported_seq":2421,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813773+0000","last_change":"2024-03-19T03:00:04.461050+0000","last_active":"2024-03-19T03:06:09.813773+0000","last_peered":"2024-03-19T03:06:09.813773+0000","last_clean":"2024-03-19T03:06:09.813773+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T03:06:09.813773+0000","last_undegraded":"2024-03-19T03:06:09.813773+0000","last_fullsized":"2024-03-19T03:06:09.813773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:49.707937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054309962000000003,"stat_sum":{"num_bytes":20068260,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":19,"num_read_kb":8201,"num_write":329,"num_write_kb":19659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"38'348","reported_seq":2448,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.813079+0000","last_change":"2024-03-19T03:00:04.518539+0000","last_active":"2024-03-19T03:06:09.813079+0000","last_peered":"2024-03-19T03:06:09.813079+0000","last_clean":"2024-03-19T03:06:09.813079+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T03:06:09.813079+0000","last_undegraded":"2024-03-19T03:06:09.813079+0000","last_fullsized":"2024-03-19T03:06:09.813079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:46.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051753202999999998,"stat_sum":{"num_bytes":6591355,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":25,"num_read_kb":4114,"num_write":347,"num_write_kb":6624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"38'331","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823940+0000","last_change":"2024-03-19T03:00:04.458228+0000","last_active":"2024-03-19T03:06:09.823940+0000","last_peered":"2024-03-19T03:06:09.823940+0000","last_clean":"2024-03-19T03:06:09.823940+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T03:06:09.823940+0000","last_undegraded":"2024-03-19T03:06:09.823940+0000","last_fullsized":"2024-03-19T03:06:09.823940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:05:00.342939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051735362,"stat_sum":{"num_bytes":12769638,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":28,"num_read_kb":4115,"num_write":331,"num_write_kb":12643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"38'343","reported_seq":2440,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.849648+0000","last_change":"2024-03-19T03:00:04.518609+0000","last_active":"2024-03-19T03:06:09.849648+0000","last_peered":"2024-03-19T03:06:09.849648+0000","last_clean":"2024-03-19T03:06:09.849648+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T03:06:09.849648+0000","last_undegraded":"2024-03-19T03:06:09.849648+0000","last_fullsized":"2024-03-19T03:06:09.849648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:41.608998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051913059999999997,"stat_sum":{"num_bytes":23638346,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":344,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"38'298","reported_seq":2388,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.795953+0000","last_change":"2024-03-19T03:00:04.268010+0000","last_active":"2024-03-19T03:06:09.795953+0000","last_peered":"2024-03-19T03:06:09.795953+0000","last_clean":"2024-03-19T03:06:09.795953+0000","last_became_active":"2024-03-19T02:55:45.392555+0000","last_became_peered":"2024-03-19T02:55:45.392555+0000","last_unstale":"2024-03-19T03:06:09.795953+0000","last_undegraded":"2024-03-19T03:06:09.795953+0000","last_fullsized":"2024-03-19T03:06:09.795953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:37.417889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052882827,"stat_sum":{"num_bytes":15124789,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":21,"num_read_kb":4106,"num_write":297,"num_write_kb":14929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"38'322","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828302+0000","last_change":"2024-03-19T03:00:04.520474+0000","last_active":"2024-03-19T03:06:09.828302+0000","last_peered":"2024-03-19T03:06:09.828302+0000","last_clean":"2024-03-19T03:06:09.828302+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T03:06:09.828302+0000","last_undegraded":"2024-03-19T03:06:09.828302+0000","last_fullsized":"2024-03-19T03:06:09.828302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:29.088176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045138699999999997,"stat_sum":{"num_bytes":8352128,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":30,"num_read_kb":4119,"num_write":322,"num_write_kb":12425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"38'304","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.757119+0000","last_change":"2024-03-19T03:00:04.469206+0000","last_active":"2024-03-19T03:06:09.757119+0000","last_peered":"2024-03-19T03:06:09.757119+0000","last_clean":"2024-03-19T03:06:09.757119+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T03:06:09.757119+0000","last_undegraded":"2024-03-19T03:06:09.757119+0000","last_fullsized":"2024-03-19T03:06:09.757119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.863780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033183580999999997,"stat_sum":{"num_bytes":17872088,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":8208,"num_write":302,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"38'305","reported_seq":2406,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.768960+0000","last_change":"2024-03-19T03:00:04.468803+0000","last_active":"2024-03-19T03:06:09.768960+0000","last_peered":"2024-03-19T03:06:09.768960+0000","last_clean":"2024-03-19T03:06:09.768960+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T03:06:09.768960+0000","last_undegraded":"2024-03-19T03:06:09.768960+0000","last_fullsized":"2024-03-19T03:06:09.768960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:05.106811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036038919000000003,"stat_sum":{"num_bytes":18621884,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":26,"num_read_kb":8210,"num_write":305,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"38'292","reported_seq":2385,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.761101+0000","last_change":"2024-03-19T03:00:04.233381+0000","last_active":"2024-03-19T03:06:09.761101+0000","last_peered":"2024-03-19T03:06:09.761101+0000","last_clean":"2024-03-19T03:06:09.761101+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T03:06:09.761101+0000","last_undegraded":"2024-03-19T03:06:09.761101+0000","last_fullsized":"2024-03-19T03:06:09.761101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:36:06.236162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016224016000000001,"stat_sum":{"num_bytes":10799086,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":18,"num_read_kb":10,"num_write":291,"num_write_kb":14795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"38'286","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.918837+0000","last_change":"2024-03-19T03:00:04.638962+0000","last_active":"2024-03-19T03:06:09.918837+0000","last_peered":"2024-03-19T03:06:09.918837+0000","last_clean":"2024-03-19T03:06:09.918837+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T03:06:09.918837+0000","last_undegraded":"2024-03-19T03:06:09.918837+0000","last_fullsized":"2024-03-19T03:06:09.918837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:12.950806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061626093999999999,"stat_sum":{"num_bytes":18869857,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":26,"num_read_kb":4115,"num_write":285,"num_write_kb":26775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"38'330","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.890035+0000","last_change":"2024-03-19T03:00:04.294731+0000","last_active":"2024-03-19T03:06:09.890035+0000","last_peered":"2024-03-19T03:06:09.890035+0000","last_clean":"2024-03-19T03:06:09.890035+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.890035+0000","last_undegraded":"2024-03-19T03:06:09.890035+0000","last_fullsized":"2024-03-19T03:06:09.890035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.987902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.06936341,"stat_sum":{"num_bytes":7021822,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":330,"num_write_kb":6980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"38'351","reported_seq":2453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.838055+0000","last_change":"2024-03-19T03:00:04.731871+0000","last_active":"2024-03-19T03:06:09.838055+0000","last_peered":"2024-03-19T03:06:09.838055+0000","last_clean":"2024-03-19T03:06:09.838055+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T03:06:09.838055+0000","last_undegraded":"2024-03-19T03:06:09.838055+0000","last_fullsized":"2024-03-19T03:06:09.838055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:26.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047989829999999997,"stat_sum":{"num_bytes":22976920,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":351,"num_write_kb":22623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"38'348","reported_seq":2450,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.868400+0000","last_change":"2024-03-19T03:00:04.463722+0000","last_active":"2024-03-19T03:06:09.868400+0000","last_peered":"2024-03-19T03:06:09.868400+0000","last_clean":"2024-03-19T03:06:09.868400+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T03:06:09.868400+0000","last_undegraded":"2024-03-19T03:06:09.868400+0000","last_fullsized":"2024-03-19T03:06:09.868400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:06.211614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054164370000000003,"stat_sum":{"num_bytes":10727184,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":27,"num_read_kb":4115,"num_write":346,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"38'344","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.771765+0000","last_change":"2024-03-19T03:00:04.463595+0000","last_active":"2024-03-19T03:06:09.771765+0000","last_peered":"2024-03-19T03:06:09.771765+0000","last_clean":"2024-03-19T03:06:09.771765+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T03:06:09.771765+0000","last_undegraded":"2024-03-19T03:06:09.771765+0000","last_fullsized":"2024-03-19T03:06:09.771765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:53:29.237872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054055032000000003,"stat_sum":{"num_bytes":10124226,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":32,"num_read_kb":4117,"num_write":343,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"38'290","reported_seq":2398,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798540+0000","last_change":"2024-03-19T03:00:04.265271+0000","last_active":"2024-03-19T03:06:09.798540+0000","last_peered":"2024-03-19T03:06:09.798540+0000","last_clean":"2024-03-19T03:06:09.798540+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T03:06:09.798540+0000","last_undegraded":"2024-03-19T03:06:09.798540+0000","last_fullsized":"2024-03-19T03:06:09.798540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:17.102471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.049230071,"stat_sum":{"num_bytes":7542726,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":289,"num_write_kb":7514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"38'322","reported_seq":2430,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828563+0000","last_change":"2024-03-19T03:00:04.518999+0000","last_active":"2024-03-19T03:06:09.828563+0000","last_peered":"2024-03-19T03:06:09.828563+0000","last_clean":"2024-03-19T03:06:09.828563+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T03:06:09.828563+0000","last_undegraded":"2024-03-19T03:06:09.828563+0000","last_fullsized":"2024-03-19T03:06:09.828563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:30.582720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050023194999999999,"stat_sum":{"num_bytes":14383098,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":32,"num_read_kb":16400,"num_write":321,"num_write_kb":18311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"38'352","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.764360+0000","last_change":"2024-03-19T03:00:04.469246+0000","last_active":"2024-03-19T03:06:09.764360+0000","last_peered":"2024-03-19T03:06:09.764360+0000","last_clean":"2024-03-19T03:06:09.764360+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T03:06:09.764360+0000","last_undegraded":"2024-03-19T03:06:09.764360+0000","last_fullsized":"2024-03-19T03:06:09.764360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:17:26.060817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033531875000000003,"stat_sum":{"num_bytes":23233989,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":33,"num_read_kb":12308,"num_write":352,"num_write_kb":31057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"38'318","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925503+0000","last_change":"2024-03-19T03:00:04.294471+0000","last_active":"2024-03-19T03:06:09.925503+0000","last_peered":"2024-03-19T03:06:09.925503+0000","last_clean":"2024-03-19T03:06:09.925503+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T03:06:09.925503+0000","last_undegraded":"2024-03-19T03:06:09.925503+0000","last_fullsized":"2024-03-19T03:06:09.925503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:35.963767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068442499000000004,"stat_sum":{"num_bytes":13700535,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":33,"num_read_kb":3602,"num_write":320,"num_write_kb":13502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"38'341","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806378+0000","last_change":"2024-03-19T03:00:04.460208+0000","last_active":"2024-03-19T03:06:09.806378+0000","last_peered":"2024-03-19T03:06:09.806378+0000","last_clean":"2024-03-19T03:06:09.806378+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T03:06:09.806378+0000","last_undegraded":"2024-03-19T03:06:09.806378+0000","last_fullsized":"2024-03-19T03:06:09.806378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:08.718707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053636608000000002,"stat_sum":{"num_bytes":9951503,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":340,"num_write_kb":13990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"38'325","reported_seq":2437,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.900508+0000","last_change":"2024-03-19T03:00:04.634734+0000","last_active":"2024-03-19T03:06:09.900508+0000","last_peered":"2024-03-19T03:06:09.900508+0000","last_clean":"2024-03-19T03:06:09.900508+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T03:06:09.900508+0000","last_undegraded":"2024-03-19T03:06:09.900508+0000","last_fullsized":"2024-03-19T03:06:09.900508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:56.724601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069022274999999994,"stat_sum":{"num_bytes":15469691,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":37,"num_read_kb":13386,"num_write":325,"num_write_kb":19374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"38'347","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.784715+0000","last_change":"2024-03-19T03:00:04.464949+0000","last_active":"2024-03-19T03:06:09.784715+0000","last_peered":"2024-03-19T03:06:09.784715+0000","last_clean":"2024-03-19T03:06:09.784715+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T03:06:09.784715+0000","last_undegraded":"2024-03-19T03:06:09.784715+0000","last_fullsized":"2024-03-19T03:06:09.784715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:40.894087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055361406000000002,"stat_sum":{"num_bytes":13836038,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":38,"num_read_kb":4325,"num_write":347,"num_write_kb":13690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"38'316","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896418+0000","last_change":"2024-03-19T03:00:04.636666+0000","last_active":"2024-03-19T03:06:09.896418+0000","last_peered":"2024-03-19T03:06:09.896418+0000","last_clean":"2024-03-19T03:06:09.896418+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T03:06:09.896418+0000","last_undegraded":"2024-03-19T03:06:09.896418+0000","last_fullsized":"2024-03-19T03:06:09.896418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:13:09.492010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066517528000000006,"stat_sum":{"num_bytes":14678298,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":314,"num_write_kb":14504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"38'357","reported_seq":2459,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.763988+0000","last_change":"2024-03-19T03:00:04.469451+0000","last_active":"2024-03-19T03:06:09.763988+0000","last_peered":"2024-03-19T03:06:09.763988+0000","last_clean":"2024-03-19T03:06:09.763988+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T03:06:09.763988+0000","last_undegraded":"2024-03-19T03:06:09.763988+0000","last_fullsized":"2024-03-19T03:06:09.763988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:08.415495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033380219000000003,"stat_sum":{"num_bytes":6267254,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":357,"num_write_kb":6310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"38'323","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.797829+0000","last_change":"2024-03-19T03:00:04.731007+0000","last_active":"2024-03-19T03:06:09.797829+0000","last_peered":"2024-03-19T03:06:09.797829+0000","last_clean":"2024-03-19T03:06:09.797829+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T03:06:09.797829+0000","last_undegraded":"2024-03-19T03:06:09.797829+0000","last_fullsized":"2024-03-19T03:06:09.797829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:38.782912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046987292,"stat_sum":{"num_bytes":20194837,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":27,"num_read_kb":4111,"num_write":325,"num_write_kb":23913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"38'312","reported_seq":2410,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.925331+0000","last_change":"2024-03-19T03:00:04.639714+0000","last_active":"2024-03-19T03:06:09.925331+0000","last_peered":"2024-03-19T03:06:09.925331+0000","last_clean":"2024-03-19T03:06:09.925331+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T03:06:09.925331+0000","last_undegraded":"2024-03-19T03:06:09.925331+0000","last_fullsized":"2024-03-19T03:06:09.925331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:07:54.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054969302999999997,"stat_sum":{"num_bytes":19046740,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":22,"num_read_kb":7383,"num_write":312,"num_write_kb":22038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"38'348","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.926885+0000","last_change":"2024-03-19T03:00:04.295066+0000","last_active":"2024-03-19T03:06:09.926885+0000","last_peered":"2024-03-19T03:06:09.926885+0000","last_clean":"2024-03-19T03:06:09.926885+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T03:06:09.926885+0000","last_undegraded":"2024-03-19T03:06:09.926885+0000","last_fullsized":"2024-03-19T03:06:09.926885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:51.825764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068635079000000002,"stat_sum":{"num_bytes":22225330,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":36,"num_read_kb":12306,"num_write":348,"num_write_kb":21889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"38'345","reported_seq":2457,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781725+0000","last_change":"2024-03-19T03:00:04.460792+0000","last_active":"2024-03-19T03:06:09.781725+0000","last_peered":"2024-03-19T03:06:09.781725+0000","last_clean":"2024-03-19T03:06:09.781725+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T03:06:09.781725+0000","last_undegraded":"2024-03-19T03:06:09.781725+0000","last_fullsized":"2024-03-19T03:06:09.781725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:11.679527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054050365000000003,"stat_sum":{"num_bytes":13922214,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":36,"num_read_kb":4118,"num_write":345,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"38'394","reported_seq":2502,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.805290+0000","last_change":"2024-03-19T03:00:04.266403+0000","last_active":"2024-03-19T03:06:09.805290+0000","last_peered":"2024-03-19T03:06:09.805290+0000","last_clean":"2024-03-19T03:06:09.805290+0000","last_became_active":"2024-03-19T02:55:45.950634+0000","last_became_peered":"2024-03-19T02:55:45.950634+0000","last_unstale":"2024-03-19T03:06:09.805290+0000","last_undegraded":"2024-03-19T03:06:09.805290+0000","last_fullsized":"2024-03-19T03:06:09.805290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:59.262092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050892488,"stat_sum":{"num_bytes":11658804,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":27,"num_read_kb":4113,"num_write":395,"num_write_kb":15629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"38'360","reported_seq":2472,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.864349+0000","last_change":"2024-03-19T03:00:04.467477+0000","last_active":"2024-03-19T03:06:09.864349+0000","last_peered":"2024-03-19T03:06:09.864349+0000","last_clean":"2024-03-19T03:06:09.864349+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T03:06:09.864349+0000","last_undegraded":"2024-03-19T03:06:09.864349+0000","last_fullsized":"2024-03-19T03:06:09.864349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:19.601901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035464137999999999,"stat_sum":{"num_bytes":11414841,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":36,"num_read_kb":4123,"num_write":359,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"38'322","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.781195+0000","last_change":"2024-03-19T03:00:04.233486+0000","last_active":"2024-03-19T03:06:09.781195+0000","last_peered":"2024-03-19T03:06:09.781195+0000","last_clean":"2024-03-19T03:06:09.781195+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T03:06:09.781195+0000","last_undegraded":"2024-03-19T03:06:09.781195+0000","last_fullsized":"2024-03-19T03:06:09.781195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.162762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016612378000000001,"stat_sum":{"num_bytes":15052974,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":23,"num_read_kb":4112,"num_write":322,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'49","reported_seq":112,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.737087+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T03:05:09.737087+0000","last_peered":"2024-03-19T03:05:09.737087+0000","last_clean":"2024-03-19T03:05:09.737087+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T03:05:09.737087+0000","last_undegraded":"2024-03-19T03:05:09.737087+0000","last_fullsized":"2024-03-19T03:05:09.737087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:43.894235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":52,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"38'306","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.922358+0000","last_change":"2024-03-19T03:00:04.295011+0000","last_active":"2024-03-19T03:06:09.922358+0000","last_peered":"2024-03-19T03:06:09.922358+0000","last_clean":"2024-03-19T03:06:09.922358+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T03:06:09.922358+0000","last_undegraded":"2024-03-19T03:06:09.922358+0000","last_fullsized":"2024-03-19T03:06:09.922358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:17.919923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066278250999999996,"stat_sum":{"num_bytes":6674098,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":15,"num_write":306,"num_write_kb":6677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"38'198","reported_seq":268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.188393+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T03:05:17.188393+0000","last_peered":"2024-03-19T03:05:17.188393+0000","last_clean":"2024-03-19T03:05:17.188393+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T03:05:17.188393+0000","last_undegraded":"2024-03-19T03:05:17.188393+0000","last_fullsized":"2024-03-19T03:05:17.188393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":61440,"num_write":198,"num_write_kb":16593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"38'29","reported_seq":87,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.904891+0000","last_change":"2024-03-19T02:54:55.192236+0000","last_active":"2024-03-19T03:05:15.904891+0000","last_peered":"2024-03-19T03:05:15.904891+0000","last_clean":"2024-03-19T03:05:15.904891+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T03:05:15.904891+0000","last_undegraded":"2024-03-19T03:05:15.904891+0000","last_fullsized":"2024-03-19T03:05:15.904891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:14.617853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":29,"num_write_kb":4131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"38'386","reported_seq":2495,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.946964+0000","last_change":"2024-03-19T03:00:04.638741+0000","last_active":"2024-03-19T03:06:09.946964+0000","last_peered":"2024-03-19T03:06:09.946964+0000","last_clean":"2024-03-19T03:06:09.946964+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T03:06:09.946964+0000","last_undegraded":"2024-03-19T03:06:09.946964+0000","last_fullsized":"2024-03-19T03:06:09.946964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:31.894668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061005943,"stat_sum":{"num_bytes":22416031,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":34,"num_read_kb":16407,"num_write":386,"num_write_kb":31278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"38'751","reported_seq":814,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:06.635700+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T03:06:06.635700+0000","last_peered":"2024-03-19T03:06:06.635700+0000","last_clean":"2024-03-19T03:06:06.635700+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T03:06:06.635700+0000","last_undegraded":"2024-03-19T03:06:06.635700+0000","last_fullsized":"2024-03-19T03:06:06.635700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:43:30.109754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":748,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"38'321","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828476+0000","last_change":"2024-03-19T03:00:04.731670+0000","last_active":"2024-03-19T03:06:09.828476+0000","last_peered":"2024-03-19T03:06:09.828476+0000","last_clean":"2024-03-19T03:06:09.828476+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T03:06:09.828476+0000","last_undegraded":"2024-03-19T03:06:09.828476+0000","last_fullsized":"2024-03-19T03:06:09.828476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:37.837633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048289845999999997,"stat_sum":{"num_bytes":20433584,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":31,"num_read_kb":16402,"num_write":319,"num_write_kb":24181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'8","reported_seq":62,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.733040+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T03:05:09.733040+0000","last_peered":"2024-03-19T03:05:09.733040+0000","last_clean":"2024-03-19T03:05:09.733040+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T03:05:09.733040+0000","last_undegraded":"2024-03-19T03:05:09.733040+0000","last_fullsized":"2024-03-19T03:05:09.733040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"38'348","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.804516+0000","last_change":"2024-03-19T03:00:04.266848+0000","last_active":"2024-03-19T03:06:09.804516+0000","last_peered":"2024-03-19T03:06:09.804516+0000","last_clean":"2024-03-19T03:06:09.804516+0000","last_became_active":"2024-03-19T02:55:45.403591+0000","last_became_peered":"2024-03-19T02:55:45.403591+0000","last_unstale":"2024-03-19T03:06:09.804516+0000","last_undegraded":"2024-03-19T03:06:09.804516+0000","last_fullsized":"2024-03-19T03:06:09.804516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:03.822452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050955334999999997,"stat_sum":{"num_bytes":22779940,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":32,"num_read_kb":10328,"num_write":348,"num_write_kb":22426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"38'1621","reported_seq":1681,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.125122+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T03:05:54.125122+0000","last_peered":"2024-03-19T03:05:54.125122+0000","last_clean":"2024-03-19T03:05:54.125122+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T03:05:54.125122+0000","last_undegraded":"2024-03-19T03:05:54.125122+0000","last_fullsized":"2024-03-19T03:05:54.125122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:32.430617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":19565,"num_write":1617,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"38'308","reported_seq":2416,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.889708+0000","last_change":"2024-03-19T03:00:04.295350+0000","last_active":"2024-03-19T03:06:09.889708+0000","last_peered":"2024-03-19T03:06:09.889708+0000","last_clean":"2024-03-19T03:06:09.889708+0000","last_became_active":"2024-03-19T02:55:45.952615+0000","last_became_peered":"2024-03-19T02:55:45.952615+0000","last_unstale":"2024-03-19T03:06:09.889708+0000","last_undegraded":"2024-03-19T03:06:09.889708+0000","last_fullsized":"2024-03-19T03:06:09.889708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:35:42.703979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067870387000000004,"stat_sum":{"num_bytes":9678845,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":28,"num_read_kb":4114,"num_write":307,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"38'258","reported_seq":327,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:39.898558+0000","last_change":"2024-03-19T02:54:55.193522+0000","last_active":"2024-03-19T03:05:39.898558+0000","last_peered":"2024-03-19T03:05:39.898558+0000","last_clean":"2024-03-19T03:05:39.898558+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T03:05:39.898558+0000","last_undegraded":"2024-03-19T03:05:39.898558+0000","last_fullsized":"2024-03-19T03:05:39.898558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:09.144197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":16407,"num_write":255,"num_write_kb":8344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"38'309","reported_seq":2407,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.759258+0000","last_change":"2024-03-19T03:00:04.233584+0000","last_active":"2024-03-19T03:06:09.759258+0000","last_peered":"2024-03-19T03:06:09.759258+0000","last_clean":"2024-03-19T03:06:09.759258+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T03:06:09.759258+0000","last_undegraded":"2024-03-19T03:06:09.759258+0000","last_fullsized":"2024-03-19T03:06:09.759258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:28.958102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016376714000000001,"stat_sum":{"num_bytes":15448536,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":23,"num_read_kb":4110,"num_write":308,"num_write_kb":19342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"38'161","reported_seq":245,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.120316+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T03:05:54.120316+0000","last_peered":"2024-03-19T03:05:54.120316+0000","last_clean":"2024-03-19T03:05:54.120316+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T03:05:54.120316+0000","last_undegraded":"2024-03-19T03:05:54.120316+0000","last_fullsized":"2024-03-19T03:05:54.120316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":57372,"num_write":154,"num_write_kb":12387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"38'321","reported_seq":2423,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.823622+0000","last_change":"2024-03-19T03:00:04.517597+0000","last_active":"2024-03-19T03:06:09.823622+0000","last_peered":"2024-03-19T03:06:09.823622+0000","last_clean":"2024-03-19T03:06:09.823622+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T03:06:09.823622+0000","last_undegraded":"2024-03-19T03:06:09.823622+0000","last_fullsized":"2024-03-19T03:06:09.823622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:01:07.022928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053512995000000001,"stat_sum":{"num_bytes":2174562,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":27,"num_read_kb":19,"num_write":323,"num_write_kb":2244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"37'154","reported_seq":228,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.015888+0000","last_change":"2024-03-19T02:54:55.198257+0000","last_active":"2024-03-19T03:05:15.015888+0000","last_peered":"2024-03-19T03:05:15.015888+0000","last_clean":"2024-03-19T03:05:15.015888+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T03:05:15.015888+0000","last_undegraded":"2024-03-19T03:05:15.015888+0000","last_fullsized":"2024-03-19T03:05:15.015888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:51.395174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":41000,"num_write":152,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"38'331","reported_seq":2443,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.896906+0000","last_change":"2024-03-19T03:00:04.293854+0000","last_active":"2024-03-19T03:06:09.896906+0000","last_peered":"2024-03-19T03:06:09.896906+0000","last_clean":"2024-03-19T03:06:09.896906+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T03:06:09.896906+0000","last_undegraded":"2024-03-19T03:06:09.896906+0000","last_fullsized":"2024-03-19T03:06:09.896906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:32.540025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068156319000000007,"stat_sum":{"num_bytes":18375725,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":37,"num_read_kb":16403,"num_write":329,"num_write_kb":26314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"37'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.450142+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T03:05:10.450142+0000","last_peered":"2024-03-19T03:05:10.450142+0000","last_clean":"2024-03-19T03:05:10.450142+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T03:05:10.450142+0000","last_undegraded":"2024-03-19T03:05:10.450142+0000","last_fullsized":"2024-03-19T03:05:10.450142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:50.061307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"38'339","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819009+0000","last_change":"2024-03-19T03:00:04.518776+0000","last_active":"2024-03-19T03:06:09.819009+0000","last_peered":"2024-03-19T03:06:09.819009+0000","last_clean":"2024-03-19T03:06:09.819009+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T03:06:09.819009+0000","last_undegraded":"2024-03-19T03:06:09.819009+0000","last_fullsized":"2024-03-19T03:06:09.819009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.456673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051813761,"stat_sum":{"num_bytes":7359660,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":339,"num_write_kb":7364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"38'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.732969+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T03:05:09.732969+0000","last_peered":"2024-03-19T03:05:09.732969+0000","last_clean":"2024-03-19T03:05:09.732969+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T03:05:09.732969+0000","last_undegraded":"2024-03-19T03:05:09.732969+0000","last_fullsized":"2024-03-19T03:05:09.732969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"38'340","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.855709+0000","last_change":"2024-03-19T03:00:04.468344+0000","last_active":"2024-03-19T03:06:09.855709+0000","last_peered":"2024-03-19T03:06:09.855709+0000","last_clean":"2024-03-19T03:06:09.855709+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T03:06:09.855709+0000","last_undegraded":"2024-03-19T03:06:09.855709+0000","last_fullsized":"2024-03-19T03:06:09.855709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:34:09.179260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035899844,"stat_sum":{"num_bytes":22036853,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":37,"num_read_kb":12302,"num_write":335,"num_write_kb":21697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"38'128","reported_seq":215,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.376836+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T03:05:38.376836+0000","last_peered":"2024-03-19T03:05:38.376836+0000","last_clean":"2024-03-19T03:05:38.376836+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T03:05:38.376836+0000","last_undegraded":"2024-03-19T03:05:38.376836+0000","last_fullsized":"2024-03-19T03:05:38.376836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:27.034870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":31075,"num_write":133,"num_write_kb":8289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"38'357","reported_seq":2465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.798019+0000","last_change":"2024-03-19T03:00:04.518831+0000","last_active":"2024-03-19T03:06:09.798019+0000","last_peered":"2024-03-19T03:06:09.798019+0000","last_clean":"2024-03-19T03:06:09.798019+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T03:06:09.798019+0000","last_undegraded":"2024-03-19T03:06:09.798019+0000","last_fullsized":"2024-03-19T03:06:09.798019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:39.124926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051911553999999999,"stat_sum":{"num_bytes":11598304,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":33,"num_read_kb":12305,"num_write":357,"num_write_kb":19699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"38'29","reported_seq":93,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.859256+0000","last_change":"2024-03-19T02:54:55.198337+0000","last_active":"2024-03-19T03:05:51.859256+0000","last_peered":"2024-03-19T03:05:51.859256+0000","last_clean":"2024-03-19T03:05:51.859256+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T03:05:51.859256+0000","last_undegraded":"2024-03-19T03:05:51.859256+0000","last_fullsized":"2024-03-19T03:05:51.859256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:57.308089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":35,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"38'296","reported_seq":2390,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.806192+0000","last_change":"2024-03-19T03:00:04.519069+0000","last_active":"2024-03-19T03:06:09.806192+0000","last_peered":"2024-03-19T03:06:09.806192+0000","last_clean":"2024-03-19T03:06:09.806192+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T03:06:09.806192+0000","last_undegraded":"2024-03-19T03:06:09.806192+0000","last_fullsized":"2024-03-19T03:06:09.806192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:27:49.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052296778000000002,"stat_sum":{"num_bytes":10391613,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":18,"num_read_kb":6155,"num_write":296,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"38'116","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:02.734657+0000","last_change":"2024-03-19T02:54:55.192437+0000","last_active":"2024-03-19T03:06:02.734657+0000","last_peered":"2024-03-19T03:06:02.734657+0000","last_clean":"2024-03-19T03:06:02.734657+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T03:06:02.734657+0000","last_undegraded":"2024-03-19T03:06:02.734657+0000","last_fullsized":"2024-03-19T03:06:02.734657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":116,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"38'323","reported_seq":2427,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.832152+0000","last_change":"2024-03-19T03:00:04.520259+0000","last_active":"2024-03-19T03:06:09.832152+0000","last_peered":"2024-03-19T03:06:09.832152+0000","last_clean":"2024-03-19T03:06:09.832152+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T03:06:09.832152+0000","last_undegraded":"2024-03-19T03:06:09.832152+0000","last_fullsized":"2024-03-19T03:06:09.832152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:46.091437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044895387000000002,"stat_sum":{"num_bytes":10001648,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":28,"num_read_kb":8211,"num_write":321,"num_write_kb":14027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"38'19","reported_seq":82,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856936+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T03:05:51.856936+0000","last_peered":"2024-03-19T03:05:51.856936+0000","last_clean":"2024-03-19T03:05:51.856936+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T03:05:51.856936+0000","last_undegraded":"2024-03-19T03:05:51.856936+0000","last_fullsized":"2024-03-19T03:05:51.856936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:16:37.076957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":20,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"38'362","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.803909+0000","last_change":"2024-03-19T03:00:04.462041+0000","last_active":"2024-03-19T03:06:09.803909+0000","last_peered":"2024-03-19T03:06:09.803909+0000","last_clean":"2024-03-19T03:06:09.803909+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T03:06:09.803909+0000","last_undegraded":"2024-03-19T03:06:09.803909+0000","last_fullsized":"2024-03-19T03:06:09.803909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:05:22.439007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055101615,"stat_sum":{"num_bytes":11152492,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":22,"num_read_kb":4110,"num_write":358,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"38'106","reported_seq":162,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.624147+0000","last_change":"2024-03-19T02:54:55.189531+0000","last_active":"2024-03-19T03:06:01.624147+0000","last_peered":"2024-03-19T03:06:01.624147+0000","last_clean":"2024-03-19T03:06:01.624147+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T03:06:01.624147+0000","last_undegraded":"2024-03-19T03:06:01.624147+0000","last_fullsized":"2024-03-19T03:06:01.624147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:32.462415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":108,"num_write_kb":4216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'329","reported_seq":2435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.828482+0000","last_change":"2024-03-19T03:00:04.517906+0000","last_active":"2024-03-19T03:06:09.828482+0000","last_peered":"2024-03-19T03:06:09.828482+0000","last_clean":"2024-03-19T03:06:09.828482+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T03:06:09.828482+0000","last_undegraded":"2024-03-19T03:06:09.828482+0000","last_fullsized":"2024-03-19T03:06:09.828482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:39.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051414362999999998,"stat_sum":{"num_bytes":15817335,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":30,"num_read_kb":4113,"num_write":329,"num_write_kb":19678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":211,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.447511+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T03:05:10.447511+0000","last_peered":"2024-03-19T03:05:10.447511+0000","last_clean":"2024-03-19T03:05:10.447511+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T03:05:10.447511+0000","last_undegraded":"2024-03-19T03:05:10.447511+0000","last_fullsized":"2024-03-19T03:05:10.447511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"38'1698","reported_seq":1756,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.922350+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T03:05:20.922350+0000","last_peered":"2024-03-19T03:05:20.922350+0000","last_clean":"2024-03-19T03:05:20.922350+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T03:05:20.922350+0000","last_undegraded":"2024-03-19T03:05:20.922350+0000","last_fullsized":"2024-03-19T03:05:20.922350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:40.316400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5356,"num_write":1694,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"38'324","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.913802+0000","last_change":"2024-03-19T03:00:04.637745+0000","last_active":"2024-03-19T03:06:09.913802+0000","last_peered":"2024-03-19T03:06:09.913802+0000","last_clean":"2024-03-19T03:06:09.913802+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T03:06:09.913802+0000","last_undegraded":"2024-03-19T03:06:09.913802+0000","last_fullsized":"2024-03-19T03:06:09.913802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:55:15.269339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063353052000000007,"stat_sum":{"num_bytes":11364310,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4110,"num_write":322,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'187","reported_seq":253,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.165979+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T03:05:15.165979+0000","last_peered":"2024-03-19T03:05:15.165979+0000","last_clean":"2024-03-19T03:05:15.165979+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T03:05:15.165979+0000","last_undegraded":"2024-03-19T03:05:15.165979+0000","last_fullsized":"2024-03-19T03:05:15.165979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":181,"num_write_kb":12402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"38'315","reported_seq":2411,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824543+0000","last_change":"2024-03-19T03:00:04.292884+0000","last_active":"2024-03-19T03:06:09.824543+0000","last_peered":"2024-03-19T03:06:09.824543+0000","last_clean":"2024-03-19T03:06:09.824543+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T03:06:09.824543+0000","last_undegraded":"2024-03-19T03:06:09.824543+0000","last_fullsized":"2024-03-19T03:06:09.824543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:15.770392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068696631999999994,"stat_sum":{"num_bytes":14176427,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":313,"num_write_kb":14010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"38'94","reported_seq":242,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.057114+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T03:05:20.057114+0000","last_peered":"2024-03-19T03:05:20.057114+0000","last_clean":"2024-03-19T03:05:20.057114+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T03:05:20.057114+0000","last_undegraded":"2024-03-19T03:05:20.057114+0000","last_fullsized":"2024-03-19T03:05:20.057114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:31.778660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":581,"num_write":90,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"38'324","reported_seq":2426,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.785089+0000","last_change":"2024-03-19T03:00:04.732387+0000","last_active":"2024-03-19T03:06:09.785089+0000","last_peered":"2024-03-19T03:06:09.785089+0000","last_clean":"2024-03-19T03:06:09.785089+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T03:06:09.785089+0000","last_undegraded":"2024-03-19T03:06:09.785089+0000","last_fullsized":"2024-03-19T03:06:09.785089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:15.401411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044152405999999998,"stat_sum":{"num_bytes":9810356,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":324,"num_write_kb":9748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'61","reported_seq":125,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.331498+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T03:05:15.331498+0000","last_peered":"2024-03-19T03:05:15.331498+0000","last_clean":"2024-03-19T03:05:15.331498+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T03:05:15.331498+0000","last_undegraded":"2024-03-19T03:05:15.331498+0000","last_fullsized":"2024-03-19T03:05:15.331498+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:36.896232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":60,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"38'358","reported_seq":2462,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.819761+0000","last_change":"2024-03-19T03:00:04.233217+0000","last_active":"2024-03-19T03:06:09.819761+0000","last_peered":"2024-03-19T03:06:09.819761+0000","last_clean":"2024-03-19T03:06:09.819761+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T03:06:09.819761+0000","last_undegraded":"2024-03-19T03:06:09.819761+0000","last_fullsized":"2024-03-19T03:06:09.819761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:01.512234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016334565999999998,"stat_sum":{"num_bytes":7144435,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":28,"num_read_kb":4112,"num_write":358,"num_write_kb":7164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"37'363","reported_seq":418,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.705301+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T03:05:16.705301+0000","last_peered":"2024-03-19T03:05:16.705301+0000","last_clean":"2024-03-19T03:05:16.705301+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T03:05:16.705301+0000","last_undegraded":"2024-03-19T03:05:16.705301+0000","last_fullsized":"2024-03-19T03:05:16.705301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:51:33.682755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":360,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'337","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.796224+0000","last_change":"2024-03-19T03:00:04.461266+0000","last_active":"2024-03-19T03:06:09.796224+0000","last_peered":"2024-03-19T03:06:09.796224+0000","last_clean":"2024-03-19T03:06:09.796224+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T03:06:09.796224+0000","last_undegraded":"2024-03-19T03:06:09.796224+0000","last_fullsized":"2024-03-19T03:06:09.796224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:10:36.694537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054510613999999999,"stat_sum":{"num_bytes":18185428,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":28,"num_read_kb":9793,"num_write":337,"num_write_kb":22033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'364","reported_seq":424,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.763732+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T03:05:16.763732+0000","last_peered":"2024-03-19T03:05:16.763732+0000","last_clean":"2024-03-19T03:05:16.763732+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T03:05:16.763732+0000","last_undegraded":"2024-03-19T03:05:16.763732+0000","last_fullsized":"2024-03-19T03:05:16.763732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":360,"num_write_kb":8331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"38'347","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.775787+0000","last_change":"2024-03-19T03:00:04.730764+0000","last_active":"2024-03-19T03:06:09.775787+0000","last_peered":"2024-03-19T03:06:09.775787+0000","last_clean":"2024-03-19T03:06:09.775787+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T03:06:09.775787+0000","last_undegraded":"2024-03-19T03:06:09.775787+0000","last_fullsized":"2024-03-19T03:06:09.775787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:38:48.638029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047353098000000003,"stat_sum":{"num_bytes":7011110,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":27,"num_read_kb":4110,"num_write":348,"num_write_kb":6996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'268","reported_seq":336,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:05.909837+0000","last_change":"2024-03-19T02:54:55.197257+0000","last_active":"2024-03-19T03:06:05.909837+0000","last_peered":"2024-03-19T03:06:05.909837+0000","last_clean":"2024-03-19T03:06:05.909837+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T03:06:05.909837+0000","last_undegraded":"2024-03-19T03:06:05.909837+0000","last_fullsized":"2024-03-19T03:06:05.909837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:21.130331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":638,"num_write":612,"num_write_kb":2118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"38'307","reported_seq":2400,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.779701+0000","last_change":"2024-03-19T03:00:04.266141+0000","last_active":"2024-03-19T03:06:09.779701+0000","last_peered":"2024-03-19T03:06:09.779701+0000","last_clean":"2024-03-19T03:06:09.779701+0000","last_became_active":"2024-03-19T02:55:45.402992+0000","last_became_peered":"2024-03-19T02:55:45.402992+0000","last_unstale":"2024-03-19T03:06:09.779701+0000","last_undegraded":"2024-03-19T03:06:09.779701+0000","last_fullsized":"2024-03-19T03:06:09.779701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:04.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050031514999999999,"stat_sum":{"num_bytes":18622985,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":24,"num_read_kb":16,"num_write":307,"num_write_kb":22442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'147","reported_seq":212,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.498499+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T03:05:40.498499+0000","last_peered":"2024-03-19T03:05:40.498499+0000","last_clean":"2024-03-19T03:05:40.498499+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T03:05:40.498499+0000","last_undegraded":"2024-03-19T03:05:40.498499+0000","last_fullsized":"2024-03-19T03:05:40.498499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":9370,"num_write":147,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"38'352","reported_seq":2467,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.888162+0000","last_change":"2024-03-19T03:00:04.295347+0000","last_active":"2024-03-19T03:06:09.888162+0000","last_peered":"2024-03-19T03:06:09.888162+0000","last_clean":"2024-03-19T03:06:09.888162+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:09.888162+0000","last_undegraded":"2024-03-19T03:06:09.888162+0000","last_fullsized":"2024-03-19T03:06:09.888162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:56:05.317379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068525034999999998,"stat_sum":{"num_bytes":35227376,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":40,"num_read_kb":16714,"num_write":347,"num_write_kb":34590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'17","reported_seq":76,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.056765+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T03:06:01.056765+0000","last_peered":"2024-03-19T03:06:01.056765+0000","last_clean":"2024-03-19T03:06:01.056765+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T03:06:01.056765+0000","last_undegraded":"2024-03-19T03:06:01.056765+0000","last_fullsized":"2024-03-19T03:06:01.056765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":20,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"38'345","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.788944+0000","last_change":"2024-03-19T03:00:04.233647+0000","last_active":"2024-03-19T03:06:09.788944+0000","last_peered":"2024-03-19T03:06:09.788944+0000","last_clean":"2024-03-19T03:06:09.788944+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T03:06:09.788944+0000","last_undegraded":"2024-03-19T03:06:09.788944+0000","last_fullsized":"2024-03-19T03:06:09.788944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:10.432157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016349961999999999,"stat_sum":{"num_bytes":23588150,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":30,"num_read_kb":20,"num_write":344,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"38'264","reported_seq":321,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.856959+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T03:05:51.856959+0000","last_peered":"2024-03-19T03:05:51.856959+0000","last_clean":"2024-03-19T03:05:51.856959+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T03:05:51.856959+0000","last_undegraded":"2024-03-19T03:05:51.856959+0000","last_fullsized":"2024-03-19T03:05:51.856959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:47.479910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":264,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"38'344","reported_seq":2454,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.943424+0000","last_change":"2024-03-19T03:00:04.639224+0000","last_active":"2024-03-19T03:06:09.943424+0000","last_peered":"2024-03-19T03:06:09.943424+0000","last_clean":"2024-03-19T03:06:09.943424+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T03:06:09.943424+0000","last_undegraded":"2024-03-19T03:06:09.943424+0000","last_fullsized":"2024-03-19T03:06:09.943424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:17.069557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061753209000000003,"stat_sum":{"num_bytes":19421231,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":34,"num_read_kb":12304,"num_write":344,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"38'74","reported_seq":133,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:38.487245+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T03:05:38.487245+0000","last_peered":"2024-03-19T03:05:38.487245+0000","last_clean":"2024-03-19T03:05:38.487245+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T03:05:38.487245+0000","last_undegraded":"2024-03-19T03:05:38.487245+0000","last_fullsized":"2024-03-19T03:05:38.487245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:08.577077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":73,"num_write_kb":8285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"38'330","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.927473+0000","last_change":"2024-03-19T03:00:04.639280+0000","last_active":"2024-03-19T03:06:09.927473+0000","last_peered":"2024-03-19T03:06:09.927473+0000","last_clean":"2024-03-19T03:06:09.927473+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T03:06:09.927473+0000","last_undegraded":"2024-03-19T03:06:09.927473+0000","last_fullsized":"2024-03-19T03:06:09.927473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:22:15.157112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061740428,"stat_sum":{"num_bytes":14078096,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":23,"num_read_kb":8205,"num_write":330,"num_write_kb":13929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'281","reported_seq":351,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.024734+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T03:05:17.024734+0000","last_peered":"2024-03-19T03:05:17.024734+0000","last_clean":"2024-03-19T03:05:17.024734+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T03:05:17.024734+0000","last_undegraded":"2024-03-19T03:05:17.024734+0000","last_fullsized":"2024-03-19T03:05:17.024734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":40970,"num_write":274,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"38'342","reported_seq":2444,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.719517+0000","last_change":"2024-03-19T03:00:04.467727+0000","last_active":"2024-03-19T03:06:09.719517+0000","last_peered":"2024-03-19T03:06:09.719517+0000","last_clean":"2024-03-19T03:06:09.719517+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T03:06:09.719517+0000","last_undegraded":"2024-03-19T03:06:09.719517+0000","last_fullsized":"2024-03-19T03:06:09.719517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:03.908237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035412037,"stat_sum":{"num_bytes":11182535,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":26,"num_read_kb":4109,"num_write":343,"num_write_kb":11053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"38'303","reported_seq":2405,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.824265+0000","last_change":"2024-03-19T03:00:04.460570+0000","last_active":"2024-03-19T03:06:09.824265+0000","last_peered":"2024-03-19T03:06:09.824265+0000","last_clean":"2024-03-19T03:06:09.824265+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T03:06:09.824265+0000","last_undegraded":"2024-03-19T03:06:09.824265+0000","last_fullsized":"2024-03-19T03:06:09.824265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:09:22.869183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053825845999999997,"stat_sum":{"num_bytes":7076024,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":303,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"37'88","reported_seq":152,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.432014+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T03:05:15.432014+0000","last_peered":"2024-03-19T03:05:15.432014+0000","last_clean":"2024-03-19T03:05:15.432014+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T03:05:15.432014+0000","last_undegraded":"2024-03-19T03:05:15.432014+0000","last_fullsized":"2024-03-19T03:05:15.432014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":92,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"38'342","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.772096+0000","last_change":"2024-03-19T03:00:04.460533+0000","last_active":"2024-03-19T03:06:09.772096+0000","last_peered":"2024-03-19T03:06:09.772096+0000","last_clean":"2024-03-19T03:06:09.772096+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T03:06:09.772096+0000","last_undegraded":"2024-03-19T03:06:09.772096+0000","last_fullsized":"2024-03-19T03:06:09.772096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:51:06.702309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053805551,"stat_sum":{"num_bytes":27334536,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":24,"num_read_kb":7741,"num_write":341,"num_write_kb":30967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"38'2315","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.196171+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T03:05:40.196171+0000","last_peered":"2024-03-19T03:05:40.196171+0000","last_clean":"2024-03-19T03:05:40.196171+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T03:05:40.196171+0000","last_undegraded":"2024-03-19T03:05:40.196171+0000","last_fullsized":"2024-03-19T03:05:40.196171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":49170,"num_write":2306,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"38'324","reported_seq":2431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.894896+0000","last_change":"2024-03-19T03:00:04.294754+0000","last_active":"2024-03-19T03:06:09.894896+0000","last_peered":"2024-03-19T03:06:09.894896+0000","last_clean":"2024-03-19T03:06:09.894896+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T03:06:09.894896+0000","last_undegraded":"2024-03-19T03:06:09.894896+0000","last_fullsized":"2024-03-19T03:06:09.894896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:06.724957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067674389000000001,"stat_sum":{"num_bytes":14508603,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":31,"num_read_kb":8211,"num_write":323,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"38'1551","reported_seq":1618,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.698465+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T03:06:01.698465+0000","last_peered":"2024-03-19T03:06:01.698465+0000","last_clean":"2024-03-19T03:06:01.698465+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T03:06:01.698465+0000","last_undegraded":"2024-03-19T03:06:01.698465+0000","last_fullsized":"2024-03-19T03:06:01.698465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:50.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":42561,"num_write":1549,"num_write_kb":13574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"38'331","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:09.899725+0000","last_change":"2024-03-19T03:00:04.636091+0000","last_active":"2024-03-19T03:06:09.899725+0000","last_peered":"2024-03-19T03:06:09.899725+0000","last_clean":"2024-03-19T03:06:09.899725+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T03:06:09.899725+0000","last_undegraded":"2024-03-19T03:06:09.899725+0000","last_fullsized":"2024-03-19T03:06:09.899725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:21:22.564399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065375344000000002,"stat_sum":{"num_bytes":15000004,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":21,"num_read_kb":8204,"num_write":332,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"37'148","reported_seq":225,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.050653+0000","last_change":"2024-03-19T02:54:55.201089+0000","last_active":"2024-03-19T03:05:20.050653+0000","last_peered":"2024-03-19T03:05:20.050653+0000","last_clean":"2024-03-19T03:05:20.050653+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T03:05:20.050653+0000","last_undegraded":"2024-03-19T03:05:20.050653+0000","last_fullsized":"2024-03-19T03:05:20.050653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:40.008818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":45076,"num_write":147,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":939458329,"num_objects":10563,"num_object_clones":0,"num_object_copies":21126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10563,"num_whiteouts":0,"num_read":1809,"num_read_kb":363000,"num_write":21147,"num_write_kb":1062305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1709027328,"data_stored":1877384053,"data_compressed":164805636,"data_compressed_allocated":182276096,"data_compressed_original":405610496,"omap_allocated":0,"internal_metadata":0},"log_size":21178,"ondisk_log_size":21178,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":227028075,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":557,"num_read_kb":727830,"num_write":11856,"num_write_kb":232365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120127488,"data_stored":454056150,"data_compressed":53210578,"data_compressed_allocated":67698688,"data_compressed_original":401686528,"omap_allocated":0,"internal_metadata":0},"log_size":11549,"ondisk_log_size":11549,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332176,"kb_used_data":242696,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94039664,"statfs":{"total":96636764160,"available":96296615936,"internally_reserved":0,"allocated":248520704,"data_stored":332228426,"data_compressed":32896566,"data_compressed_allocated":38154240,"data_compressed_original":130297856,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283912,"kb_used_data":194496,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94087928,"statfs":{"total":96636764160,"available":96346038272,"internally_reserved":0,"allocated":199163904,"data_stored":260529012,"data_compressed":19584348,"data_compressed_allocated":23072768,"data_compressed_original":89657344,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364880,"kb_used_data":253192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94006960,"statfs":{"total":96636764160,"available":96263127040,"internally_reserved":0,"allocated":259268608,"data_stored":316361128,"data_compressed":23587682,"data_compressed_allocated":27353088,"data_compressed_original":92348416,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320448,"kb_used_data":253176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94051392,"statfs":{"total":96636764160,"available":96308625408,"internally_reserved":0,"allocated":259252224,"data_stored":332900836,"data_compressed":37042510,"data_compressed_allocated":41844736,"data_compressed_original":123842560,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336116,"kb_used_data":273004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94035724,"statfs":{"total":96636764160,"available":96292581376,"internally_reserved":0,"allocated":279556096,"data_stored":335606622,"data_compressed":26219015,"data_compressed_allocated":29872128,"data_compressed_original":93212672,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182320,"kb_used_data":137256,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94189520,"statfs":{"total":96636764160,"available":96450068480,"internally_reserved":0,"allocated":140550144,"data_stored":170537803,"data_compressed":15586883,"data_compressed_allocated":17731584,"data_compressed_original":51924992,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208528,"kb_used_data":163464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94163312,"statfs":{"total":96636764160,"available":96423231488,"internally_reserved":0,"allocated":167387136,"data_stored":233785712,"data_compressed":26814662,"data_compressed_allocated":30638080,"data_compressed_original":101830656,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326524,"kb_used_data":259252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94045316,"statfs":{"total":96636764160,"available":96302403584,"internally_reserved":0,"allocated":265474048,"data_stored":327658768,"data_compressed":31314452,"data_compressed_allocated":35168256,"data_compressed_original":104353792,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12947456,"data_stored":50332286,"data_compressed":6253471,"data_compressed_allocated":7770112,"data_compressed_original":45166592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":58720416,"data_compressed":7340199,"data_compressed_allocated":9289728,"data_compressed_original":54624256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":21505709,"data_compressed":2290904,"data_compressed_allocated":3022848,"data_compressed_original":17223680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11042816,"data_stored":46672205,"data_compressed":5800482,"data_compressed_allocated":7323648,"data_compressed_original":42962944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14008320,"data_stored":62914686,"data_compressed":7417000,"data_compressed_allocated":9461760,"data_compressed_original":58380288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":62914650,"data_compressed":7062565,"data_compressed_allocated":8916992,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20590592,"data_stored":71303840,"data_compressed":7958306,"data_compressed_allocated":10170368,"data_compressed_original":60891136,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":79692358,"data_compressed":9087651,"data_compressed_allocated":11743232,"data_compressed_original":71311360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256462848,"data_stored":282593685,"data_compressed":26461690,"data_compressed_allocated":29118464,"data_compressed_original":63418368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156610560,"data_stored":179655935,"data_compressed":20569111,"data_compressed_allocated":22691840,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135131136,"data_stored":152463714,"data_compressed":14071312,"data_compressed_allocated":15671296,"data_compressed_original":37679104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270573568,"data_stored":292209323,"data_compressed":21139415,"data_compressed_allocated":23465984,"data_compressed_original":52903936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245436416,"data_stored":273102592,"data_compressed":30129180,"data_compressed_allocated":33001472,"data_compressed_original":68767744,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238751744,"data_stored":256562920,"data_compressed":17028787,"data_compressed_allocated":19054592,"data_compressed_original":44527616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178155520,"data_stored":189041622,"data_compressed":11629622,"data_compressed_allocated":12906496,"data_compressed_original":28774400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":227905536,"data_stored":251754262,"data_compressed":23776519,"data_compressed_allocated":26365952,"data_compressed_original":58413056,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:06:14.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:06:15.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.070+0000 7fd0a3000640 1 Processor -- start 2024-03-19T03:06:15.071 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.070+0000 7fd0a3000640 1 -- start start 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.071+0000 7fd0a3000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c071190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.071+0000 7fd0a3000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd09c0c60d0 0x7fd09c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a3000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c15b740 con 0x7fd09c154dc0 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a3000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd09c15ba20 con 0x7fd09c0c60d0 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a3000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c0c8400 con 0x7fd09c072380 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a1000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd09c0c60d0 0x7fd09c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd093e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c071190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd09be00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd09c154dc0 0x7fd09c071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:54508/0 (socket says 172.21.15.92:54508) 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd09be00640 1 -- 172.21.15.92:0/1985639554 learned_addr learned my addr 172.21.15.92:0/1985639554 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a1000640 1 -- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 msgr2=0x7fd09c071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a1000640 1 --2- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c071190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a1000640 1 -- 172.21.15.92:0/1985639554 >> v1:172.21.15.159:6789/0 conn(0x7fd09c154dc0 legacy=0x7fd09c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.072+0000 7fd0a1000640 1 -- 172.21.15.92:0/1985639554 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c0c86e0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.073+0000 7fd0a1000640 1 --2- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c0c6c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd094009d90 tx=0x7fd09402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d15bff1c9be0f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:15.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.073+0000 7fd09b400640 1 -- 172.21.15.92:0/1985639554 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094039070 con 0x7fd09c0c60d0 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.073+0000 7fd09b400640 1 -- 172.21.15.92:0/1985639554 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09402cbc0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd09b400640 1 -- 172.21.15.92:0/1985639554 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094034440 con 0x7fd09c0c60d0 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 -- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 msgr2=0x7fd09c0c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c0c6c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd094009d90 tx=0x7fd09402be10 comp rx=0 tx=0).stop 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 -- 172.21.15.92:0/1985639554 shutdown_connections 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1985639554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c0c6c90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 -- 172.21.15.92:0/1985639554 >> 172.21.15.92:0/1985639554 conn(0x7fd09c0c8d90 msgr2=0x7fd09c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:15.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 -- 172.21.15.92:0/1985639554 shutdown_connections 2024-03-19T03:06:15.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.074+0000 7fd0a3000640 1 -- 172.21.15.92:0/1985639554 wait complete. 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.076+0000 7fd0a3000640 1 Processor -- start 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.076+0000 7fd0a3000640 1 -- start start 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c154dc0 0x7fd09c1f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c0c7350 con 0x7fd09c154dc0 2024-03-19T03:06:15.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c0c71d0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a3000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c0c74d0 con 0x7fd09c072380 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd09be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd093e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd093e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39618/0 (socket says 172.21.15.92:39618) 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd09be00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:42142/0 (socket says 172.21.15.92:42142) 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd093e00640 1 -- 172.21.15.92:0/1351394808 learned_addr learned my addr 172.21.15.92:0/1351394808 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.077+0000 7fd0a1000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c154dc0 0x7fd09c1f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 -- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 msgr2=0x7fd09c156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 -- 172.21.15.92:0/1351394808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c154dc0 msgr2=0x7fd09c1f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c154dc0 0x7fd09c1f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 -- 172.21.15.92:0/1351394808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c1e4050 con 0x7fd09c0c60d0 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd093e00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd08400e9b0 tx=0x7fd08400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084017070 con 0x7fd09c0c60d0 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08400c9a0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c1559b0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08400cd90 con 0x7fd09c0c60d0 2024-03-19T03:06:15.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.078+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c1e5ad0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.080+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09c0684c0 con 0x7fd09c0c60d0 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.080+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd08401a070 con 0x7fd09c0c60d0 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.081+0000 7fd098c00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 0x7fd07806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.081+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fd08408ac30 con 0x7fd09c0c60d0 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.082+0000 7fd09be00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 0x7fd07806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.082+0000 7fd09be00640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 0x7fd07806e0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd08c005880 tx=0x7fd08c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:15.088 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.088+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd084058e90 con 0x7fd09c0c60d0 2024-03-19T03:06:15.289 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.289+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd09c0c5d50 con 0x7fd09c0c60d0 2024-03-19T03:06:15.292 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.291+0000 7fd098c00640 1 -- 172.21.15.92:0/1351394808 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+17584 (secure 0 0 0) 0x7fd08405cf30 con 0x7fd09c0c60d0 2024-03-19T03:06:15.292 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:15.292 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":38,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T03:05:09.726374+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6839/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6838/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6837/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6838/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6837/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6835/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/1521694689":"2024-03-20T02:57:04.595375+0000","172.21.15.159:6835/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6838/1521694689":"2024-03-20T02:57:04.595375+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:06:15.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.293+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 msgr2=0x7fd07806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.293+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 0x7fd07806e0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd08c005880 tx=0x7fd08c009040 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.293+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 msgr2=0x7fd09c1f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.293+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd08400e9b0 tx=0x7fd08400ee70 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 shutdown_connections 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd07806bc00 0x7fd07806e0c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c154dc0 0x7fd09c1f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd09c0c60d0 0x7fd09c1f4140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 --2- 172.21.15.92:0/1351394808 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd09c072380 0x7fd09c156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:15.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 >> 172.21.15.92:0/1351394808 conn(0x7fd09c0c8d90 msgr2=0x7fd09c06e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:15.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 shutdown_connections 2024-03-19T03:06:15.296 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:15.294+0000 7fd0a3000640 1 -- 172.21.15.92:0/1351394808 wait complete. 2024-03-19T03:06:16.315 INFO:tasks.ceph:Scrubbing osd.0 2024-03-19T03:06:16.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:16.532 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.531+0000 7f104ca00640 1 Processor -- start 2024-03-19T03:06:16.532 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.531+0000 7f104ca00640 1 -- start start 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f104ca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 0x7f1048157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f104ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1048071a70 0x7f104806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f104ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1048073320 con 0x7f10481512a0 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f104ca00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1048072150 con 0x7f1048071a70 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f104ca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f10480c8290 con 0x7f1048151680 2024-03-19T03:06:16.534 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f1045400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 0x7f1048157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f1046800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1048071a70 0x7f104806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f1045e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f10481512a0 0x7f1048072c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53292/0 (socket says 172.21.15.92:53292) 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.533+0000 7f1045e00640 1 -- 172.21.15.92:0/3596981788 learned_addr learned my addr 172.21.15.92:0/3596981788 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.534+0000 7f1045400640 1 -- 172.21.15.92:0/3596981788 >> v2:172.21.15.92:3300/0 conn(0x7f1048071a70 msgr2=0x7f104806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.534+0000 7f1045400640 1 --2- 172.21.15.92:0/3596981788 >> v2:172.21.15.92:3300/0 conn(0x7f1048071a70 0x7f104806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.534+0000 7f1045400640 1 -- 172.21.15.92:0/3596981788 >> v1:172.21.15.92:6789/0 conn(0x7f10481512a0 legacy=0x7f1048072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.535 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.534+0000 7f1045400640 1 -- 172.21.15.92:0/3596981788 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10480c8570 con 0x7f1048151680 2024-03-19T03:06:16.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f1045400640 1 --2- 172.21.15.92:0/3596981788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 0x7f1048157bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f10300097a0 tx=0x7f103002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9f8b475e673d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f1044a00640 1 -- 172.21.15.92:0/3596981788 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030034070 con 0x7f1048151680 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f1044a00640 1 -- 172.21.15.92:0/3596981788 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1030030800 con 0x7f1048151680 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f1044a00640 1 -- 172.21.15.92:0/3596981788 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030030bd0 con 0x7f1048151680 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f104ca00640 1 -- 172.21.15.92:0/3596981788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 msgr2=0x7f1048157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.535+0000 7f104ca00640 1 --2- 172.21.15.92:0/3596981788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 0x7f1048157bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f10300097a0 tx=0x7f103002b940 comp rx=0 tx=0).stop 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 -- 172.21.15.92:0/3596981788 shutdown_connections 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 --2- 172.21.15.92:0/3596981788 >> v2:172.21.15.92:3300/0 conn(0x7f1048071a70 0x7f104806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 --2- 172.21.15.92:0/3596981788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048151680 0x7f1048157bf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 -- 172.21.15.92:0/3596981788 >> 172.21.15.92:0/3596981788 conn(0x7f10480cc1f0 msgr2=0x7f10480cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 -- 172.21.15.92:0/3596981788 shutdown_connections 2024-03-19T03:06:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.536+0000 7f104ca00640 1 -- 172.21.15.92:0/3596981788 wait complete. 2024-03-19T03:06:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.537+0000 7f104ca00640 1 Processor -- start 2024-03-19T03:06:16.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.538+0000 7f104ca00640 1 -- start start 2024-03-19T03:06:16.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.538+0000 7f104ca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 0x7f1048142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.538+0000 7f104ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10481512a0 0x7f1048141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.540 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f104ca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1048151680 0x7f1048141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f104ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10480c6af0 con 0x7f10481512a0 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f104ca00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f10480726f0 con 0x7f1048151680 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f104ca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f104806bf00 con 0x7f1048071a70 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 0x7f1048142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 0x7f1048142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36220/0 (socket says 172.21.15.92:36220) 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 -- 172.21.15.92:0/2186227505 learned_addr learned my addr 172.21.15.92:0/2186227505 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045400640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10481512a0 0x7f1048141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1046800640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1048151680 0x7f1048141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 -- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1048151680 msgr2=0x7f1048141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.541 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1048151680 0x7f1048141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 -- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10481512a0 msgr2=0x7f1048141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10481512a0 0x7f1048141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1048146530 con 0x7f1048071a70 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.539+0000 7f1045e00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 0x7f1048142750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f103800cb10 tx=0x7f103800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.540+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038017070 con 0x7f1048071a70 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.540+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103800ac70 con 0x7f1048071a70 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.540+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038012620 con 0x7f1048071a70 2024-03-19T03:06:16.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.540+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1048144620 con 0x7f1048071a70 2024-03-19T03:06:16.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.540+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10480caed0 con 0x7f1048071a70 2024-03-19T03:06:16.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.541+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10480684c0 con 0x7f1048071a70 2024-03-19T03:06:16.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.542+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f103801a070 con 0x7f1048071a70 2024-03-19T03:06:16.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f103e000640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f102406bbd0 0x7f102406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f103808aad0 con 0x7f1048071a70 2024-03-19T03:06:16.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f1045400640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f102406bbd0 0x7f102406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f103e000640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f10240742d0 0x7f1024076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1024076e40 con 0x7f10240742d0 2024-03-19T03:06:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f103808f2a0 con 0x7f1048071a70 2024-03-19T03:06:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.543+0000 7f1046800640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f10240742d0 0x7f1024076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.544+0000 7f1045400640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f102406bbd0 0x7f102406e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1030036000 tx=0x7f103003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.544+0000 7f1046800640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f10240742d0 0x7f1024076770 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.545+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== osd.0 v2:172.21.15.92:6814/3305119790 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1024076e40 con 0x7f10240742d0 2024-03-19T03:06:16.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.559+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 --> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f10480c77f0 con 0x7f10240742d0 2024-03-19T03:06:16.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f103e000640 1 -- 172.21.15.92:0/2186227505 <== osd.0 v2:172.21.15.92:6814/3305119790 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f10480c77f0 con 0x7f10240742d0 2024-03-19T03:06:16.560 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:16.561 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:16.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f10240742d0 msgr2=0x7f1024076770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.92:6814/3305119790,v1:172.21.15.92:6817/3305119790] conn(0x7f10240742d0 0x7f1024076770 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f102406bbd0 msgr2=0x7f102406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f102406bbd0 0x7f102406e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1030036000 tx=0x7f103003b040 comp rx=0 tx=0).stop 2024-03-19T03:06:16.562 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 msgr2=0x7f1048142750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 --2- 172.21.15.92:0/2186227505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1048071a70 0x7f1048142750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f103800cb10 tx=0x7f103800a060 comp rx=0 tx=0).stop 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f1046800640 1 -- 172.21.15.92:0/2186227505 reap_dead start 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 shutdown_connections 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.560+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 >> 172.21.15.92:0/2186227505 conn(0x7f10480cc1f0 msgr2=0x7f10480c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.561+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 shutdown_connections 2024-03-19T03:06:16.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.561+0000 7f104ca00640 1 -- 172.21.15.92:0/2186227505 wait complete. 2024-03-19T03:06:16.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-19T03:06:16.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.778+0000 7f626a000640 1 Processor -- start 2024-03-19T03:06:16.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.778+0000 7f626a000640 1 -- start start 2024-03-19T03:06:16.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.779+0000 7f626a000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62641526a0 0x7f6264158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.779+0000 7f626a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6264152a80 0x7f62640c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.779+0000 7f626a000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640c6d30 con 0x7f62640c6460 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.779+0000 7f626a000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f62640c6ff0 con 0x7f6264152a80 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.779+0000 7f626a000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f62640c72d0 con 0x7f62641526a0 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f625be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6264152a80 0x7f62640c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f6269600640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f62640c6460 0x7f626406cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55812/0 (socket says 172.21.15.92:55812) 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f6268c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62641526a0 0x7f6264158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.781 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f6269600640 1 -- 172.21.15.92:0/354374534 learned_addr learned my addr 172.21.15.92:0/354374534 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:16.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f625be00640 1 -- 172.21.15.92:0/354374534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62641526a0 msgr2=0x7f6264158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f625be00640 1 --2- 172.21.15.92:0/354374534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62641526a0 0x7f6264158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f6263e00640 1 -- 172.21.15.92:0/354374534 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584186635 0 0) 0x7f62640c6d30 con 0x7f62640c6460 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f625be00640 1 -- 172.21.15.92:0/354374534 >> v1:172.21.15.159:6789/0 conn(0x7f62640c6460 legacy=0x7f626406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.780+0000 7f625be00640 1 -- 172.21.15.92:0/354374534 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62640c43d0 con 0x7f6264152a80 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.781+0000 7f625be00640 1 --2- 172.21.15.92:0/354374534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62640c3790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f62540097a0 tx=0x7f625402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6e4f5a207effec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f6263e00640 1 -- 172.21.15.92:0/354374534 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254034070 con 0x7f6264152a80 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f6263e00640 1 -- 172.21.15.92:0/354374534 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6254030800 con 0x7f6264152a80 2024-03-19T03:06:16.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f6263e00640 1 -- 172.21.15.92:0/354374534 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254030bd0 con 0x7f6264152a80 2024-03-19T03:06:16.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 -- 172.21.15.92:0/354374534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 msgr2=0x7f62640c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 --2- 172.21.15.92:0/354374534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62640c3790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f62540097a0 tx=0x7f625402b940 comp rx=0 tx=0).stop 2024-03-19T03:06:16.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 -- 172.21.15.92:0/354374534 shutdown_connections 2024-03-19T03:06:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 --2- 172.21.15.92:0/354374534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62640c3790 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 --2- 172.21.15.92:0/354374534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62641526a0 0x7f6264158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.782+0000 7f626a000640 1 -- 172.21.15.92:0/354374534 >> 172.21.15.92:0/354374534 conn(0x7f626406bac0 msgr2=0x7f62640cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.783+0000 7f626a000640 1 -- 172.21.15.92:0/354374534 shutdown_connections 2024-03-19T03:06:16.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.783+0000 7f626a000640 1 -- 172.21.15.92:0/354374534 wait complete. 2024-03-19T03:06:16.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.784+0000 7f626a000640 1 Processor -- start 2024-03-19T03:06:16.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.784+0000 7f626a000640 1 -- start start 2024-03-19T03:06:16.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 0x7f62640cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.787 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62641efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f62640c4080 con 0x7f6264152a80 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6264158d40 con 0x7f62641526a0 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.785+0000 7f626a000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f62640c3cd0 con 0x7f62640c6460 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f625be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 0x7f62640cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36242/0 (socket says 172.21.15.92:36242) 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 -- 172.21.15.92:0/908107189 learned_addr learned my addr 172.21.15.92:0/908107189 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:16.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f625be00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 0x7f62640cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39630/0 (socket says 172.21.15.92:39630) 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6269600640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62641efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 -- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 msgr2=0x7f62640cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 0x7f62640cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 -- 172.21.15.92:0/908107189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 msgr2=0x7f62641efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62641efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 -- 172.21.15.92:0/908107189 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6264152e80 con 0x7f62640c6460 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6268c00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f624c00e9b0 tx=0x7f624c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c017070 con 0x7f62640c6460 2024-03-19T03:06:16.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.786+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f624c00c9a0 con 0x7f62640c6460 2024-03-19T03:06:16.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.787+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6264153140 con 0x7f62640c6460 2024-03-19T03:06:16.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.787+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c00cd90 con 0x7f62640c6460 2024-03-19T03:06:16.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.787+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62641f36b0 con 0x7f62640c6460 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.788+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626404f580 con 0x7f62640c6460 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.789+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f624c01a070 con 0x7f62640c6460 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.789+0000 7f6261600640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 0x7f624006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.790+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f624c08ada0 con 0x7f62640c6460 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.790+0000 7f625be00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 0x7f624006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.790+0000 7f625be00640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 0x7f624006e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6254036000 tx=0x7f6254008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.794+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f624c058ed0 con 0x7f62640c6460 2024-03-19T03:06:16.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.990+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f62641f3e10 con 0x7f624006bbd0 2024-03-19T03:06:16.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.991+0000 7f6261600640 1 -- 172.21.15.92:0/908107189 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f62641f3e10 con 0x7f624006bbd0 2024-03-19T03:06:16.997 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 0 to deep-scrub 2024-03-19T03:06:16.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 msgr2=0x7f624006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 0x7f624006e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6254036000 tx=0x7f6254008030 comp rx=0 tx=0).stop 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 msgr2=0x7f6264155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f624c00e9b0 tx=0x7f624c00ee70 comp rx=0 tx=0).stop 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 shutdown_connections 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f624006bbd0 0x7f624006e090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6264152a80 0x7f62641efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62641526a0 0x7f62640cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 --2- 172.21.15.92:0/908107189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62640c6460 0x7f6264155ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 >> 172.21.15.92:0/908107189 conn(0x7f626406bac0 msgr2=0x7f626406d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 shutdown_connections 2024-03-19T03:06:16.999 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:16.994+0000 7f626a000640 1 -- 172.21.15.92:0/908107189 wait complete. 2024-03-19T03:06:17.015 INFO:tasks.ceph:Scrubbing osd.1 2024-03-19T03:06:17.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:17.228 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.227+0000 7ff92fa00640 1 Processor -- start 2024-03-19T03:06:17.229 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.227+0000 7ff92fa00640 1 -- start start 2024-03-19T03:06:17.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.228+0000 7ff92fa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 0x7ff92815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.230 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.228+0000 7ff92fa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff9280c7b80 0x7ff928154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.228+0000 7ff92fa00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92815f2b0 con 0x7ff92806b890 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.228+0000 7ff92fa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff9280c8af0 con 0x7ff9280c7b80 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.228+0000 7ff92fa00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280c74e0 con 0x7ff92806b4b0 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92d000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 0x7ff92815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff927e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff92806b890 0x7ff92815b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55824/0 (socket says 172.21.15.92:55824) 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff927e00640 1 -- 172.21.15.92:0/2685634649 learned_addr learned my addr 172.21.15.92:0/2685634649 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:17.231 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92da00640 1 --2- 172.21.15.92:0/2685634649 >> v2:172.21.15.92:3300/0 conn(0x7ff9280c7b80 0x7ff928154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92d000640 1 -- 172.21.15.92:0/2685634649 >> v2:172.21.15.92:3300/0 conn(0x7ff9280c7b80 msgr2=0x7ff928154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:17.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92d000640 1 --2- 172.21.15.92:0/2685634649 >> v2:172.21.15.92:3300/0 conn(0x7ff9280c7b80 0x7ff928154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.232 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92d000640 1 -- 172.21.15.92:0/2685634649 >> v1:172.21.15.159:6789/0 conn(0x7ff92806b890 legacy=0x7ff92815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.229+0000 7ff92d000640 1 -- 172.21.15.92:0/2685634649 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9280c6f20 con 0x7ff92806b4b0 2024-03-19T03:06:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.230+0000 7ff92d000640 1 --2- 172.21.15.92:0/2685634649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 0x7ff92815ed70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0097d0 tx=0x7ff91c02b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e620716d9f6f9057 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.231+0000 7ff927400640 1 -- 172.21.15.92:0/2685634649 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c037070 con 0x7ff92806b4b0 2024-03-19T03:06:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.231+0000 7ff927400640 1 -- 172.21.15.92:0/2685634649 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91c030710 con 0x7ff92806b4b0 2024-03-19T03:06:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.231+0000 7ff927400640 1 -- 172.21.15.92:0/2685634649 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c030ae0 con 0x7ff92806b4b0 2024-03-19T03:06:17.234 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.231+0000 7ff92fa00640 1 -- 172.21.15.92:0/2685634649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 msgr2=0x7ff92815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.231+0000 7ff92fa00640 1 --2- 172.21.15.92:0/2685634649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 0x7ff92815ed70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0097d0 tx=0x7ff91c02b6e0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 -- 172.21.15.92:0/2685634649 shutdown_connections 2024-03-19T03:06:17.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 --2- 172.21.15.92:0/2685634649 >> v2:172.21.15.92:3300/0 conn(0x7ff9280c7b80 0x7ff928154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 --2- 172.21.15.92:0/2685634649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff92806b4b0 0x7ff92815ed70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 -- 172.21.15.92:0/2685634649 >> 172.21.15.92:0/2685634649 conn(0x7ff9280cc1f0 msgr2=0x7ff9280cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 -- 172.21.15.92:0/2685634649 shutdown_connections 2024-03-19T03:06:17.236 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.232+0000 7ff92fa00640 1 -- 172.21.15.92:0/2685634649 wait complete. 2024-03-19T03:06:17.238 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.233+0000 7ff92fa00640 1 Processor -- start 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.233+0000 7ff92fa00640 1 -- start start 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff92806b4b0 0x7ff9281f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 0x7ff9280c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9280c7b80 0x7ff9281f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff92815b220 con 0x7ff92806b890 2024-03-19T03:06:17.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff92815af40 con 0x7ff92806b4b0 2024-03-19T03:06:17.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.234+0000 7ff92fa00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280719a0 con 0x7ff9280c7b80 2024-03-19T03:06:17.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff927e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 0x7ff9280c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff92806b4b0 0x7ff9281f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.241 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff927e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 0x7ff9280c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47412/0 (socket says 172.21.15.92:47412) 2024-03-19T03:06:17.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff927e00640 1 -- 172.21.15.92:0/3602007525 learned_addr learned my addr 172.21.15.92:0/3602007525 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:17.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9280c7b80 0x7ff9281f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 -- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9280c7b80 msgr2=0x7ff9281f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.242 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9280c7b80 0x7ff9281f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 -- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 msgr2=0x7ff9280c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 0x7ff9280c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92815c140 con 0x7ff92806b4b0 2024-03-19T03:06:17.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff927e00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff92806b890 0x7ff9280c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.235+0000 7ff92d000640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff92806b4b0 0x7ff9281f4f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff91c032000 tx=0x7ff91c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.236+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c041020 con 0x7ff92806b4b0 2024-03-19T03:06:17.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.236+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91c03c870 con 0x7ff92806b4b0 2024-03-19T03:06:17.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.236+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c037040 con 0x7ff92806b4b0 2024-03-19T03:06:17.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.236+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92815a1c0 con 0x7ff92806b4b0 2024-03-19T03:06:17.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.236+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92815aba0 con 0x7ff92806b4b0 2024-03-19T03:06:17.245 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.237+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9280684c0 con 0x7ff92806b4b0 2024-03-19T03:06:17.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.238+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff91c044020 con 0x7ff92806b4b0 2024-03-19T03:06:17.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.238+0000 7ff924c00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff90806bbd0 0x7ff90806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7ff91c0ac4f0 con 0x7ff92806b4b0 2024-03-19T03:06:17.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff927e00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff90806bbd0 0x7ff90806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff924c00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7ff9080742d0 0x7ff908076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff91c030710 con 0x7ff9080742d0 2024-03-19T03:06:17.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff91c0b3050 con 0x7ff92806b4b0 2024-03-19T03:06:17.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff92da00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7ff9080742d0 0x7ff908076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.239+0000 7ff927e00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff90806bbd0 0x7ff90806e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff9281eec30 tx=0x7ff914000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.240+0000 7ff92da00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7ff9080742d0 0x7ff908076770 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.241+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== osd.1 v2:172.21.15.92:6808/3121211588 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff91c030710 con 0x7ff9080742d0 2024-03-19T03:06:17.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.255+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 --> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff9280c3350 con 0x7ff9080742d0 2024-03-19T03:06:17.257 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.255+0000 7ff924c00640 1 -- 172.21.15.92:0/3602007525 <== osd.1 v2:172.21.15.92:6808/3121211588 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff9280c3350 con 0x7ff9080742d0 2024-03-19T03:06:17.258 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:17.259 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:17.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7ff9080742d0 msgr2=0x7ff908076770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.92:6808/3121211588,v1:172.21.15.92:6809/3121211588] conn(0x7ff9080742d0 0x7ff908076770 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff90806bbd0 msgr2=0x7ff90806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff90806bbd0 0x7ff90806e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff9281eec30 tx=0x7ff914000bc0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff92806b4b0 msgr2=0x7ff9281f4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 --2- 172.21.15.92:0/3602007525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff92806b4b0 0x7ff9281f4f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff91c032000 tx=0x7ff91c0046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92da00640 1 -- 172.21.15.92:0/3602007525 reap_dead start 2024-03-19T03:06:17.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 shutdown_connections 2024-03-19T03:06:17.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 >> 172.21.15.92:0/3602007525 conn(0x7ff9280cc1f0 msgr2=0x7ff9281525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 shutdown_connections 2024-03-19T03:06:17.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.256+0000 7ff92fa00640 1 -- 172.21.15.92:0/3602007525 wait complete. 2024-03-19T03:06:17.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-19T03:06:17.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.483+0000 7f87a4400640 1 Processor -- start 2024-03-19T03:06:17.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.483+0000 7f87a4400640 1 -- start start 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a4400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c0c6470 0x7f879c0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a4400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a4400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879c0c9a80 con 0x7f879c152bf0 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a4400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f879c0c6c90 con 0x7f879c0c6470 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a4400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f879c159910 con 0x7f879c1593d0 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a1000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c0c6470 0x7f879c0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a1000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c0c6470 0x7f879c0c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47420/0 (socket says 172.21.15.92:47420) 2024-03-19T03:06:17.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a1000640 1 -- 172.21.15.92:0/374849781 learned_addr learned my addr 172.21.15.92:0/374849781 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:17.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.485+0000 7f87a2400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.486+0000 7f87a1000640 1 -- 172.21.15.92:0/374849781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 msgr2=0x7f879c0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.486+0000 7f87a1000640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c0c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.486+0000 7f87a1000640 1 -- 172.21.15.92:0/374849781 >> v1:172.21.15.92:6789/0 conn(0x7f879c152bf0 legacy=0x7f879c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.486+0000 7f87a1000640 1 -- 172.21.15.92:0/374849781 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c0c8650 con 0x7f879c0c6470 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.486+0000 7f87a2400640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c0c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.487+0000 7f87a1000640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0c6470 0x7f879c0c3f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8788009870 tx=0x7f878802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917373771f77a623 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.487+0000 7f879be00640 1 -- 172.21.15.92:0/374849781 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788035020 con 0x7f879c0c6470 2024-03-19T03:06:17.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.487+0000 7f879be00640 1 -- 172.21.15.92:0/374849781 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8788031640 con 0x7f879c0c6470 2024-03-19T03:06:17.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.487+0000 7f879be00640 1 -- 172.21.15.92:0/374849781 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878802d040 con 0x7f879c0c6470 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 -- 172.21.15.92:0/374849781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0c6470 msgr2=0x7f879c0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0c6470 0x7f879c0c3f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8788009870 tx=0x7f878802b8f0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 -- 172.21.15.92:0/374849781 shutdown_connections 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c0c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 --2- 172.21.15.92:0/374849781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0c6470 0x7f879c0c3f80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 -- 172.21.15.92:0/374849781 >> 172.21.15.92:0/374849781 conn(0x7f879c0c8d90 msgr2=0x7f879c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 -- 172.21.15.92:0/374849781 shutdown_connections 2024-03-19T03:06:17.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.488+0000 7f87a4400640 1 -- 172.21.15.92:0/374849781 wait complete. 2024-03-19T03:06:17.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.490+0000 7f87a4400640 1 Processor -- start 2024-03-19T03:06:17.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.490+0000 7f87a4400640 1 -- start start 2024-03-19T03:06:17.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 0x7f879c149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.494 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f879c06ea60 con 0x7f879c152bf0 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f879c06e4a0 con 0x7f879c0c6470 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a4400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f879c06e6f0 con 0x7f879c1593d0 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47426/0 (socket says 172.21.15.92:47426) 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1000640 1 -- 172.21.15.92:0/3799362508 learned_addr learned my addr 172.21.15.92:0/3799362508 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 0x7f879c149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a2400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 -- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 msgr2=0x7f879c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c06efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 -- 172.21.15.92:0/3799362508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 msgr2=0x7f879c1ef110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1a00640 1 -- 172.21.15.92:0/3799362508 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c1f1d80 con 0x7f879c0c6470 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a2400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.491+0000 7f87a1000640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f87a1a00640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 0x7f879c149910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f879000c340 tx=0x7f879000c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87900052d0 con 0x7f879c0c6470 2024-03-19T03:06:17.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8790005490 con 0x7f879c0c6470 2024-03-19T03:06:17.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790007260 con 0x7f879c0c6470 2024-03-19T03:06:17.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879c1f2060 con 0x7f879c0c6470 2024-03-19T03:06:17.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.492+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879c1f2620 con 0x7f879c0c6470 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.493+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f879c14b7f0 con 0x7f879c0c6470 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.494+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8790007400 con 0x7f879c0c6470 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.495+0000 7f8799600640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 0x7f877c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.495+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f8790089ff0 con 0x7f879c0c6470 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.495+0000 7f87a1000640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 0x7f877c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.498 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.496+0000 7f87a1000640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 0x7f877c06e0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8788009870 tx=0x7f878803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.501 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.499+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f87900581f0 con 0x7f879c0c6470 2024-03-19T03:06:17.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.699+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f879c0684c0 con 0x7f877c06bc00 2024-03-19T03:06:17.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.700+0000 7f8799600640 1 -- 172.21.15.92:0/3799362508 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f879c0684c0 con 0x7f877c06bc00 2024-03-19T03:06:17.702 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 1 to deep-scrub 2024-03-19T03:06:17.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 msgr2=0x7f877c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 0x7f877c06e0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8788009870 tx=0x7f878803c040 comp rx=0 tx=0).stop 2024-03-19T03:06:17.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 msgr2=0x7f879c149910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.705 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 0x7f879c149910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f879000c340 tx=0x7f879000c740 comp rx=0 tx=0).stop 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 shutdown_connections 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f877c06bc00 0x7f877c06e0c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f879c1593d0 0x7f879c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c152bf0 0x7f879c1ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 --2- 172.21.15.92:0/3799362508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f879c0c6470 0x7f879c149910 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 >> 172.21.15.92:0/3799362508 conn(0x7f879c0c8d90 msgr2=0x7f879c0c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.703+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 shutdown_connections 2024-03-19T03:06:17.706 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.704+0000 7f87a4400640 1 -- 172.21.15.92:0/3799362508 wait complete. 2024-03-19T03:06:17.723 INFO:tasks.ceph:Scrubbing osd.2 2024-03-19T03:06:17.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:17.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.930+0000 7f86f6e00640 1 Processor -- start 2024-03-19T03:06:17.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.931+0000 7f86f6e00640 1 -- start start 2024-03-19T03:06:17.934 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f6e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f6e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 0x7f86f0154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f6e00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f00c6f20 con 0x7f86f006b4b0 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f6e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f86f00c8af0 con 0x7f86f00c7b80 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f6e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f86f00c74e0 con 0x7f86f006b890 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86efe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f86f006b4b0 0x7f86f015ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55828/0 (socket says 172.21.15.92:55828) 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86f4e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 0x7f86f0154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.935 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86efe00640 1 -- 172.21.15.92:0/3047982450 learned_addr learned my addr 172.21.15.92:0/3047982450 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.932+0000 7f86e7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86ef400640 1 -- 172.21.15.92:0/3047982450 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679965128 0 0) 0x7f86f00c6f20 con 0x7f86f006b4b0 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86ef400640 1 -- 172.21.15.92:0/3047982450 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d4003520 con 0x7f86f006b4b0 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86e7400640 1 -- 172.21.15.92:0/3047982450 >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 msgr2=0x7f86f0154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86e7400640 1 --2- 172.21.15.92:0/3047982450 >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 0x7f86f0154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86e7400640 1 -- 172.21.15.92:0/3047982450 >> v1:172.21.15.159:6789/0 conn(0x7f86f006b4b0 legacy=0x7f86f015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86e7400640 1 -- 172.21.15.92:0/3047982450 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f00c7200 con 0x7f86f006b890 2024-03-19T03:06:17.936 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.933+0000 7f86f4e00640 1 --2- 172.21.15.92:0/3047982450 >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 0x7f86f0154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.934+0000 7f86e7400640 1 --2- 172.21.15.92:0/3047982450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f015b740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f86d80097a0 tx=0x7f86d802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c96c8e03c610369c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.934+0000 7f86ef400640 1 -- 172.21.15.92:0/3047982450 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8035020 con 0x7f86f006b890 2024-03-19T03:06:17.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.934+0000 7f86ef400640 1 -- 172.21.15.92:0/3047982450 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d8031810 con 0x7f86f006b890 2024-03-19T03:06:17.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.934+0000 7f86ef400640 1 -- 172.21.15.92:0/3047982450 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8031be0 con 0x7f86f006b890 2024-03-19T03:06:17.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 -- 172.21.15.92:0/3047982450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 msgr2=0x7f86f015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 --2- 172.21.15.92:0/3047982450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f015b740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f86d80097a0 tx=0x7f86d802b8b0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 -- 172.21.15.92:0/3047982450 shutdown_connections 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 --2- 172.21.15.92:0/3047982450 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f015b740 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 --2- 172.21.15.92:0/3047982450 >> v2:172.21.15.159:3300/0 conn(0x7f86f00c7b80 0x7f86f0154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 -- 172.21.15.92:0/3047982450 >> 172.21.15.92:0/3047982450 conn(0x7f86f00cc1f0 msgr2=0x7f86f00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 -- 172.21.15.92:0/3047982450 shutdown_connections 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.935+0000 7f86f6e00640 1 -- 172.21.15.92:0/3047982450 wait complete. 2024-03-19T03:06:17.939 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.937+0000 7f86f6e00640 1 Processor -- start 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.937+0000 7f86f6e00640 1 -- start start 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 0x7f86f00c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f00c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86f00c7b80 0x7f86f01f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f86f015bc80 con 0x7f86f006b4b0 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f86f015b220 con 0x7f86f00c7b80 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f6e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f86f00719a0 con 0x7f86f006b890 2024-03-19T03:06:17.942 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 0x7f86f00c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f00c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 0x7f86f00c6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47434/0 (socket says 172.21.15.92:47434) 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86efe00640 1 -- 172.21.15.92:0/283927786 learned_addr learned my addr 172.21.15.92:0/283927786 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86f4e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86f00c7b80 0x7f86f01f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 -- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86f00c7b80 msgr2=0x7f86f01f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86f00c7b80 0x7f86f01f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 -- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 msgr2=0x7f86f00c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 0x7f86f00c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.943 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86e7400640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f00c2530 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.938+0000 7f86efe00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86f006b4b0 0x7f86f00c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86e7400640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f00c1270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f86d8033000 tx=0x7f86d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8041070 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d8031e40 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f00c2810 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8035040 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.939+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f01f0c00 con 0x7f86f006b890 2024-03-19T03:06:17.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.940+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86f00684c0 con 0x7f86f006b890 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.941+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f86d804c060 con 0x7f86f006b890 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86ecc00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f86cc06bbd0 0x7f86cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f86d80ac5f0 con 0x7f86f006b890 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86efe00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f86cc06bbd0 0x7f86cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86ecc00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7f86cc0742d0 0x7f86cc076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86d802eea0 con 0x7f86cc0742d0 2024-03-19T03:06:17.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f86d80b3050 con 0x7f86f006b890 2024-03-19T03:06:17.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.942+0000 7f86f4e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7f86cc0742d0 0x7f86cc076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.943+0000 7f86efe00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f86cc06bbd0 0x7f86cc06e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f86e0005880 tx=0x7f86e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.943+0000 7f86f4e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7f86cc0742d0 0x7f86cc076770 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.947 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.944+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== osd.2 v2:172.21.15.92:6800/652641961 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86d802eea0 con 0x7f86cc0742d0 2024-03-19T03:06:17.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.958+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 --> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f86f00c3110 con 0x7f86cc0742d0 2024-03-19T03:06:17.960 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.958+0000 7f86ecc00640 1 -- 172.21.15.92:0/283927786 <== osd.2 v2:172.21.15.92:6800/652641961 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f86f00c3110 con 0x7f86cc0742d0 2024-03-19T03:06:17.961 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:17.961 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:17.961 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7f86cc0742d0 msgr2=0x7f86cc076770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.92:6800/652641961,v1:172.21.15.92:6801/652641961] conn(0x7f86cc0742d0 0x7f86cc076770 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f86cc06bbd0 msgr2=0x7f86cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f86cc06bbd0 0x7f86cc06e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f86e0005880 tx=0x7f86e0009040 comp rx=0 tx=0).stop 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 msgr2=0x7f86f00c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 --2- 172.21.15.92:0/283927786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86f006b890 0x7f86f00c1270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f86d8033000 tx=0x7f86d80046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f4e00640 1 -- 172.21.15.92:0/283927786 reap_dead start 2024-03-19T03:06:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 shutdown_connections 2024-03-19T03:06:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 >> 172.21.15.92:0/283927786 conn(0x7f86f00cc1f0 msgr2=0x7f86f0152500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 shutdown_connections 2024-03-19T03:06:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:17.959+0000 7f86f6e00640 1 -- 172.21.15.92:0/283927786 wait complete. 2024-03-19T03:06:17.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-19T03:06:18.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.177+0000 7f565da00640 1 Processor -- start 2024-03-19T03:06:18.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.177+0000 7f565da00640 1 -- start start 2024-03-19T03:06:18.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.178+0000 7f565da00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 0x7f56580c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f565da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f565da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 0x7f56580c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.181 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f565da00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f56580c64e0 con 0x7f565815aaf0 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f565da00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f5658155250 con 0x7f56580c7fa0 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f565da00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56580c8ae0 con 0x7f5658071740 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 0x7f56580c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36288/0 (socket says 172.21.15.92:36288) 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 -- 172.21.15.92:0/2369147184 learned_addr learned my addr 172.21.15.92:0/2369147184 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5657400640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 0x7f56580c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.182 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 -- 172.21.15.92:0/2369147184 >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 msgr2=0x7f56580c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 0x7f56580c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 -- 172.21.15.92:0/2369147184 >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 msgr2=0x7f56580c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 0x7f56580c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656000640 1 -- 172.21.15.92:0/2369147184 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565815e180 con 0x7f5658071740 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5656a00640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 0x7f56580c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:18.183 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.179+0000 7f5657400640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 0x7f56580c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.181+0000 7f5656000640 1 --2- 172.21.15.92:0/2369147184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56580cc8a0 tx=0x7f564402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78655cdfabd53c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.181+0000 7f5655600640 1 -- 172.21.15.92:0/2369147184 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644034070 con 0x7f5658071740 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.181+0000 7f5655600640 1 -- 172.21.15.92:0/2369147184 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5644030860 con 0x7f5658071740 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.181+0000 7f5655600640 1 -- 172.21.15.92:0/2369147184 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644030c30 con 0x7f5658071740 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 -- 172.21.15.92:0/2369147184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 msgr2=0x7f56580c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 --2- 172.21.15.92:0/2369147184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56580cc8a0 tx=0x7f564402b9a0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 -- 172.21.15.92:0/2369147184 shutdown_connections 2024-03-19T03:06:18.185 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.92:3300/0 conn(0x7f565815aaf0 0x7f56580c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 --2- 172.21.15.92:0/2369147184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f56580c7340 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 --2- 172.21.15.92:0/2369147184 >> v2:172.21.15.159:3300/0 conn(0x7f56580c7fa0 0x7f56580c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 -- 172.21.15.92:0/2369147184 >> 172.21.15.92:0/2369147184 conn(0x7f565806d330 msgr2=0x7f565806d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 -- 172.21.15.92:0/2369147184 shutdown_connections 2024-03-19T03:06:18.186 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.182+0000 7f565da00640 1 -- 172.21.15.92:0/2369147184 wait complete. 2024-03-19T03:06:18.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.184+0000 7f565da00640 1 Processor -- start 2024-03-19T03:06:18.189 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.184+0000 7f565da00640 1 -- start start 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56580c7fa0 0x7f56581e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 0x7f56581f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f565815e700 con 0x7f56580c7fa0 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f565815e420 con 0x7f565815aaf0 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f565da00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56580c7880 con 0x7f5658071740 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5657400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 0x7f56581f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36300/0 (socket says 172.21.15.92:36300) 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 -- 172.21.15.92:0/1536097542 learned_addr learned my addr 172.21.15.92:0/1536097542 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656000640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56580c7fa0 0x7f56581e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 -- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 msgr2=0x7f56581f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 0x7f56581f2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 -- 172.21.15.92:0/1536097542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56580c7fa0 msgr2=0x7f56581e3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56580c7fa0 0x7f56581e3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5656a00640 1 -- 172.21.15.92:0/1536097542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565815bf80 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.185+0000 7f5657400640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 0x7f56581f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f5656a00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f564800e980 tx=0x7f564800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648017070 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f564800ca00 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564800cdd0 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565815c220 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.186+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f565815c4e0 con 0x7f5658071740 2024-03-19T03:06:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.187+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56580684c0 con 0x7f5658071740 2024-03-19T03:06:18.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.188+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f564801a070 con 0x7f5658071740 2024-03-19T03:06:18.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.189+0000 7f564ea00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 0x7f563806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.193 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.189+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f564808ac10 con 0x7f5658071740 2024-03-19T03:06:18.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.189+0000 7f5656000640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 0x7f563806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.194 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.190+0000 7f5656000640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 0x7f563806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f5644008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.193+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5648058d40 con 0x7f5658071740 2024-03-19T03:06:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.393+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f56580c7d80 con 0x7f563806bbd0 2024-03-19T03:06:18.395 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.394+0000 7f564ea00640 1 -- 172.21.15.92:0/1536097542 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f56580c7d80 con 0x7f563806bbd0 2024-03-19T03:06:18.396 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 2 to deep-scrub 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.396+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 msgr2=0x7f563806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.396+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 0x7f563806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f5644008030 comp rx=0 tx=0).stop 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.396+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 msgr2=0x7f565814c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.396+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f564800e980 tx=0x7f564800ee40 comp rx=0 tx=0).stop 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 shutdown_connections 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f563806bbd0 0x7f563806e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f565815aaf0 0x7f56581f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56580c7fa0 0x7f56581e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 --2- 172.21.15.92:0/1536097542 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5658071740 0x7f565814c1a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 >> 172.21.15.92:0/1536097542 conn(0x7f565806d330 msgr2=0x7f5658158b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 shutdown_connections 2024-03-19T03:06:18.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.397+0000 7f565da00640 1 -- 172.21.15.92:0/1536097542 wait complete. 2024-03-19T03:06:18.416 INFO:tasks.ceph:Scrubbing osd.3 2024-03-19T03:06:18.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:18.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.631+0000 7f7bb4e00640 1 Processor -- start 2024-03-19T03:06:18.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.631+0000 7f7bb4e00640 1 -- start start 2024-03-19T03:06:18.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.632+0000 7f7bb4e00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb014e4c0 0x7f7bb00c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.632+0000 7f7bb4e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bb006b950 0x7f7bb0072f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.632+0000 7f7bb4e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bb0157c70 con 0x7f7bb0072b40 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.632+0000 7f7bb4e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb0158880 con 0x7f7bb006b950 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.632+0000 7f7bb4e00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb0157f50 con 0x7f7bb014e4c0 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb014e4c0 0x7f7bb00c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb014e4c0 0x7f7bb00c3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39658/0 (socket says 172.21.15.92:39658) 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 -- 172.21.15.92:0/1930201563 learned_addr learned my addr 172.21.15.92:0/1930201563 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:18.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bae800640 1 --2- 172.21.15.92:0/1930201563 >> v2:172.21.15.92:3300/0 conn(0x7f7bb006b950 0x7f7bb0072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 -- 172.21.15.92:0/1930201563 >> v2:172.21.15.92:3300/0 conn(0x7f7bb006b950 msgr2=0x7f7bb0072f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 --2- 172.21.15.92:0/1930201563 >> v2:172.21.15.92:3300/0 conn(0x7f7bb006b950 0x7f7bb0072f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 -- 172.21.15.92:0/1930201563 >> v1:172.21.15.92:6789/0 conn(0x7f7bb0072b40 legacy=0x7f7bb006b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.633+0000 7f7bade00640 1 -- 172.21.15.92:0/1930201563 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb0158230 con 0x7f7bb014e4c0 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.634+0000 7f7bade00640 1 --2- 172.21.15.92:0/1930201563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb014e4c0 0x7f7bb00c3a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006c140 tx=0x7f7ba002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1024d995862bda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.634+0000 7f7bad400640 1 -- 172.21.15.92:0/1930201563 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0034070 con 0x7f7bb014e4c0 2024-03-19T03:06:18.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bad400640 1 -- 172.21.15.92:0/1930201563 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba0030860 con 0x7f7bb014e4c0 2024-03-19T03:06:18.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bad400640 1 -- 172.21.15.92:0/1930201563 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0030c30 con 0x7f7bb014e4c0 2024-03-19T03:06:18.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 -- 172.21.15.92:0/1930201563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb014e4c0 msgr2=0x7f7bb00c3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/1930201563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb014e4c0 0x7f7bb00c3a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006c140 tx=0x7f7ba002b9a0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 -- 172.21.15.92:0/1930201563 shutdown_connections 2024-03-19T03:06:18.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/1930201563 >> v2:172.21.15.92:3300/0 conn(0x7f7bb006b950 0x7f7bb0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/1930201563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb014e4c0 0x7f7bb00c3a80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 -- 172.21.15.92:0/1930201563 >> 172.21.15.92:0/1930201563 conn(0x7f7bb00cc1f0 msgr2=0x7f7bb00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 -- 172.21.15.92:0/1930201563 shutdown_connections 2024-03-19T03:06:18.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.635+0000 7f7bb4e00640 1 -- 172.21.15.92:0/1930201563 wait complete. 2024-03-19T03:06:18.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.637+0000 7f7bb4e00640 1 Processor -- start 2024-03-19T03:06:18.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.638+0000 7f7bb4e00640 1 -- start start 2024-03-19T03:06:18.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 0x7f7bb0157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 0x7f7bb01f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb014e4c0 0x7f7bb01f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb014ea20 con 0x7f7bb014e4c0 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb014e8a0 con 0x7f7bb0072b40 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bb4e00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb0071ba0 con 0x7f7bb006b950 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 0x7f7bb01f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 0x7f7bb0157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bae800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb014e4c0 0x7f7bb01f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 0x7f7bb0157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36310/0 (socket says 172.21.15.92:36310) 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 -- 172.21.15.92:0/3532945105 learned_addr learned my addr 172.21.15.92:0/3532945105 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 0x7f7bb01f0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39662/0 (socket says 172.21.15.92:39662) 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 -- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 msgr2=0x7f7bb01f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 0x7f7bb01f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 -- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb014e4c0 msgr2=0x7f7bb01f0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb014e4c0 0x7f7bb01f0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7bade00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb0141f30 con 0x7f7bb006b950 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.639+0000 7f7ba5400640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb0072b40 0x7f7bb01f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:18.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7bade00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 0x7f7bb0157280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ba002b9e0 tx=0x7f7ba00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0034040 con 0x7f7bb006b950 2024-03-19T03:06:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba0039d90 con 0x7f7bb006b950 2024-03-19T03:06:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0043610 con 0x7f7bb006b950 2024-03-19T03:06:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb0142210 con 0x7f7bb006b950 2024-03-19T03:06:18.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.640+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb01427d0 con 0x7f7bb006b950 2024-03-19T03:06:18.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.641+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7bb00684c0 con 0x7f7bb006b950 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.642+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ba004a060 con 0x7f7bb006b950 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.643+0000 7f7ba6a00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b8c06bbd0 0x7f7b8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.643+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f7ba00ad380 con 0x7f7bb006b950 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.643+0000 7f7ba5400640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b8c06bbd0 0x7f7b8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7ba6a00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7f7b8c0742d0 0x7f7b8c076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ba002ce90 con 0x7f7b8c0742d0 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7ba00ad730 con 0x7f7bb006b950 2024-03-19T03:06:18.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7bae800640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7f7b8c0742d0 0x7f7b8c076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7ba5400640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b8c06bbd0 0x7f7b8c06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006b760 tx=0x7f7b98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.644+0000 7f7bae800640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7f7b8c0742d0 0x7f7b8c076770 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.645+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== osd.3 v2:172.21.15.92:6818/1598837651 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ba002ce90 con 0x7f7b8c0742d0 2024-03-19T03:06:18.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.660+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 --> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7bb00c8a20 con 0x7f7b8c0742d0 2024-03-19T03:06:18.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.660+0000 7f7ba6a00640 1 -- 172.21.15.92:0/3532945105 <== osd.3 v2:172.21.15.92:6818/1598837651 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7bb00c8a20 con 0x7f7b8c0742d0 2024-03-19T03:06:18.663 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:18.664 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:18.664 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.660+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7f7b8c0742d0 msgr2=0x7f7b8c076770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.660+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.92:6818/1598837651,v1:172.21.15.92:6819/1598837651] conn(0x7f7b8c0742d0 0x7f7b8c076770 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b8c06bbd0 msgr2=0x7f7b8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b8c06bbd0 0x7f7b8c06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006b760 tx=0x7f7b98009040 comp rx=0 tx=0).stop 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 msgr2=0x7f7bb0157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.665 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 --2- 172.21.15.92:0/3532945105 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb006b950 0x7f7bb0157280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ba002b9e0 tx=0x7f7ba00046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bae800640 1 -- 172.21.15.92:0/3532945105 reap_dead start 2024-03-19T03:06:18.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 shutdown_connections 2024-03-19T03:06:18.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 >> 172.21.15.92:0/3532945105 conn(0x7f7bb00cc1f0 msgr2=0x7f7bb006f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 shutdown_connections 2024-03-19T03:06:18.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.661+0000 7f7bb4e00640 1 -- 172.21.15.92:0/3532945105 wait complete. 2024-03-19T03:06:18.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-19T03:06:18.881 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.879+0000 7fbb07400640 1 Processor -- start 2024-03-19T03:06:18.882 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.879+0000 7fbb07400640 1 -- start start 2024-03-19T03:06:18.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.880+0000 7fbb07400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbb00152bf0 0x7fbb0006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.880+0000 7fbb07400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 0x7fbb000c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.880+0000 7fbb07400640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb000c9a80 con 0x7fbb001593d0 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.880+0000 7fbb07400640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fbb000c6c90 con 0x7fbb00152bf0 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.880+0000 7fbb07400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbb00159910 con 0x7fbb000c6470 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb04a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbb00152bf0 0x7fbb0006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb05400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbb001593d0 0x7fbb000c44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55832/0 (socket says 172.21.15.92:55832) 2024-03-19T03:06:18.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb05400640 1 -- 172.21.15.92:0/2721987056 learned_addr learned my addr 172.21.15.92:0/2721987056 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbaffe00640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 0x7fbb000c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbaff400640 1 -- 172.21.15.92:0/2721987056 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108838656 0 0) 0x7fbb000c9a80 con 0x7fbb001593d0 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbaff400640 1 -- 172.21.15.92:0/2721987056 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae4003520 con 0x7fbb001593d0 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb04a00640 1 -- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 msgr2=0x7fbb000c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb04a00640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 0x7fbb000c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb04a00640 1 -- 172.21.15.92:0/2721987056 >> v1:172.21.15.159:6789/0 conn(0x7fbb001593d0 legacy=0x7fbb000c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbb04a00640 1 -- 172.21.15.92:0/2721987056 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb000c8650 con 0x7fbb00152bf0 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.881+0000 7fbaffe00640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 0x7fbb000c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:18.891 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.882+0000 7fbb04a00640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb00152bf0 0x7fbb0006eef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009d90 tx=0x7fbaf002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b612c5e03cc4117 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.882+0000 7fbaff400640 1 -- 172.21.15.92:0/2721987056 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0039070 con 0x7fbb00152bf0 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbaff400640 1 -- 172.21.15.92:0/2721987056 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf002cbc0 con 0x7fbb00152bf0 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbaff400640 1 -- 172.21.15.92:0/2721987056 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0034440 con 0x7fbb00152bf0 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 -- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb00152bf0 msgr2=0x7fbb0006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb00152bf0 0x7fbb0006eef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009d90 tx=0x7fbaf002be10 comp rx=0 tx=0).stop 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 -- 172.21.15.92:0/2721987056 shutdown_connections 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb000c6470 0x7fbb000c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 --2- 172.21.15.92:0/2721987056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb00152bf0 0x7fbb0006eef0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 -- 172.21.15.92:0/2721987056 >> 172.21.15.92:0/2721987056 conn(0x7fbb000c8d90 msgr2=0x7fbb000c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 -- 172.21.15.92:0/2721987056 shutdown_connections 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.883+0000 7fbb07400640 1 -- 172.21.15.92:0/2721987056 wait complete. 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.885+0000 7fbb07400640 1 Processor -- start 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.885+0000 7fbb07400640 1 -- start start 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb000c6470 0x7fbb001f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb001593d0 0x7fbb000cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0006ea60 con 0x7fbb000c6470 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0006e4a0 con 0x7fbb001593d0 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb07400640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0006e6f0 con 0x7fbb00152bf0 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36336/0 (socket says 172.21.15.92:36336) 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 -- 172.21.15.92:0/2513313291 learned_addr learned my addr 172.21.15.92:0/2513313291 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb04a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb000c6470 0x7fbb001f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbb05400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb001593d0 0x7fbb000cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 -- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb001593d0 msgr2=0x7fbb000cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb001593d0 0x7fbb000cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 -- 172.21.15.92:0/2513313291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb000c6470 msgr2=0x7fbb001f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb000c6470 0x7fbb001f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.886+0000 7fbaffe00640 1 -- 172.21.15.92:0/2513313291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb00072940 con 0x7fbb00152bf0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbaffe00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00ebe0 tx=0x7fbaec00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec017070 con 0x7fbb00152bf0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec00cc30 con 0x7fbb00152bf0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec013470 con 0x7fbb00152bf0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb001f48f0 con 0x7fbb00152bf0 2024-03-19T03:06:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.887+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb001f0690 con 0x7fbb00152bf0 2024-03-19T03:06:18.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.888+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb001f0480 con 0x7fbb00152bf0 2024-03-19T03:06:18.896 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.889+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbaec01a070 con 0x7fbb00152bf0 2024-03-19T03:06:18.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.890+0000 7fbafcc00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 0x7fbae006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.890+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fbaec08afe0 con 0x7fbb00152bf0 2024-03-19T03:06:18.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.890+0000 7fbb04a00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 0x7fbae006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.891+0000 7fbb04a00640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 0x7fbae006e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:18.894+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbb00072940 con 0x7fbb00152bf0 2024-03-19T03:06:19.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.092+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fbb000684c0 con 0x7fbae006bbd0 2024-03-19T03:06:19.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.093+0000 7fbafcc00640 1 -- 172.21.15.92:0/2513313291 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbb000684c0 con 0x7fbae006bbd0 2024-03-19T03:06:19.095 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 3 to deep-scrub 2024-03-19T03:06:19.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.095+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 msgr2=0x7fbae006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.095+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 0x7fbae006e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf0030000 comp rx=0 tx=0).stop 2024-03-19T03:06:19.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.095+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 msgr2=0x7fbb00071390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.095+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00ebe0 tx=0x7fbaec00c520 comp rx=0 tx=0).stop 2024-03-19T03:06:19.098 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 shutdown_connections 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fbae006bbd0 0x7fbae006e090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb001593d0 0x7fbb000cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb00152bf0 0x7fbb00071390 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 --2- 172.21.15.92:0/2513313291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb000c6470 0x7fbb001f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 >> 172.21.15.92:0/2513313291 conn(0x7fbb000c8d90 msgr2=0x7fbb0006edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 shutdown_connections 2024-03-19T03:06:19.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.096+0000 7fbb07400640 1 -- 172.21.15.92:0/2513313291 wait complete. 2024-03-19T03:06:19.116 INFO:tasks.ceph:Scrubbing osd.4 2024-03-19T03:06:19.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:19.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.338+0000 7ff31b200640 1 Processor -- start 2024-03-19T03:06:19.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.339+0000 7ff31b200640 1 -- start start 2024-03-19T03:06:19.342 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff31b200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff31b200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 0x7ff3140cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff31b200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff31415e6d0 con 0x7ff314154d50 2024-03-19T03:06:19.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff31b200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7ff3140c59b0 con 0x7ff3140caec0 2024-03-19T03:06:19.343 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff31b200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff31415e9b0 con 0x7ff314155130 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff30b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff30b400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36352/0 (socket says 172.21.15.92:36352) 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff30b400640 1 -- 172.21.15.92:0/1468249659 learned_addr learned my addr 172.21.15.92:0/1468249659 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.341+0000 7ff319200640 1 --2- 172.21.15.92:0/1468249659 >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 0x7ff3140cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.342+0000 7ff30b400640 1 -- 172.21.15.92:0/1468249659 >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 msgr2=0x7ff3140cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.342+0000 7ff30b400640 1 --2- 172.21.15.92:0/1468249659 >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 0x7ff3140cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.344 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.342+0000 7ff30b400640 1 -- 172.21.15.92:0/1468249659 >> v1:172.21.15.92:6789/0 conn(0x7ff314154d50 legacy=0x7ff31415b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.342+0000 7ff30b400640 1 -- 172.21.15.92:0/1468249659 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31415ec90 con 0x7ff314155130 2024-03-19T03:06:19.345 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.342+0000 7ff319200640 1 --2- 172.21.15.92:0/1468249659 >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 0x7ff3140cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:06:19.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.343+0000 7ff30b400640 1 --2- 172.21.15.92:0/1468249659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009870 tx=0x7ff2fc02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f134c35bb94bfac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.343+0000 7ff313400640 1 -- 172.21.15.92:0/1468249659 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc02f830 con 0x7ff314155130 2024-03-19T03:06:19.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.343+0000 7ff313400640 1 -- 172.21.15.92:0/1468249659 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2fc02f9d0 con 0x7ff314155130 2024-03-19T03:06:19.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.343+0000 7ff313400640 1 -- 172.21.15.92:0/1468249659 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc02fda0 con 0x7ff314155130 2024-03-19T03:06:19.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 -- 172.21.15.92:0/1468249659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 msgr2=0x7ff3140cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 --2- 172.21.15.92:0/1468249659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009870 tx=0x7ff2fc02ba10 comp rx=0 tx=0).stop 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 -- 172.21.15.92:0/1468249659 shutdown_connections 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 --2- 172.21.15.92:0/1468249659 >> v2:172.21.15.159:3300/0 conn(0x7ff3140caec0 0x7ff3140cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 --2- 172.21.15.92:0/1468249659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff3140cb7c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 -- 172.21.15.92:0/1468249659 >> 172.21.15.92:0/1468249659 conn(0x7ff314072970 msgr2=0x7ff314066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 -- 172.21.15.92:0/1468249659 shutdown_connections 2024-03-19T03:06:19.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.344+0000 7ff31b200640 1 -- 172.21.15.92:0/1468249659 wait complete. 2024-03-19T03:06:19.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.346+0000 7ff31b200640 1 Processor -- start 2024-03-19T03:06:19.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.346+0000 7ff31b200640 1 -- start start 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3140caec0 0x7ff314150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 0x7ff314151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff31406c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff31415e330 con 0x7ff3140caec0 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff31415e1b0 con 0x7ff314154d50 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.347+0000 7ff31b200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff31415e4b0 con 0x7ff314155130 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff313e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3140caec0 0x7ff314150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.350 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff319200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff31406c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 0x7ff314151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff319200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff31406c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36366/0 (socket says 172.21.15.92:36366) 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 0x7ff314151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39714/0 (socket says 172.21.15.92:39714) 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff319200640 1 -- 172.21.15.92:0/706773648 learned_addr learned my addr 172.21.15.92:0/706773648 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 -- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 msgr2=0x7ff31406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff314155130 0x7ff31406c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 -- 172.21.15.92:0/706773648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3140caec0 msgr2=0x7ff314150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3140caec0 0x7ff314150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.351 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.348+0000 7ff30b400640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31415a930 con 0x7ff314154d50 2024-03-19T03:06:19.352 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff30b400640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 0x7ff314151d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc033000 tx=0x7ff2fc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc037dd0 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2fc046070 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc041640 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3141538f0 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.349+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31415dcd0 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.350+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff3140684c0 con 0x7ff314154d50 2024-03-19T03:06:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.351+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff2fc040070 con 0x7ff314154d50 2024-03-19T03:06:19.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff310c00640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff2f006bbd0 0x7ff2f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.354 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7ff2fc0ad1e0 con 0x7ff314154d50 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff313e00640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff2f006bbd0 0x7ff2f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff310c00640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7ff2f00742e0 0x7ff2f0076780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc0067f0 con 0x7ff2f00742e0 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.352+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff2fc0b4050 con 0x7ff314154d50 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.353+0000 7ff319200640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7ff2f00742e0 0x7ff2f0076780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.355 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.353+0000 7ff313e00640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff2f006bbd0 0x7ff2f006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff304001780 tx=0x7ff304009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.353+0000 7ff319200640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7ff2f00742e0 0x7ff2f0076780 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.356 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.354+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== osd.4 v2:172.21.15.159:6808/4013217604 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc0067f0 con 0x7ff2f00742e0 2024-03-19T03:06:19.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.374+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 --> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff314158810 con 0x7ff2f00742e0 2024-03-19T03:06:19.376 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff310c00640 1 -- 172.21.15.92:0/706773648 <== osd.4 v2:172.21.15.159:6808/4013217604 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff314158810 con 0x7ff2f00742e0 2024-03-19T03:06:19.377 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:19.377 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:19.377 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7ff2f00742e0 msgr2=0x7ff2f0076780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6808/4013217604,v1:172.21.15.159:6809/4013217604] conn(0x7ff2f00742e0 0x7ff2f0076780 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff2f006bbd0 msgr2=0x7ff2f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7ff2f006bbd0 0x7ff2f006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff304001780 tx=0x7ff304009040 comp rx=0 tx=0).stop 2024-03-19T03:06:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 msgr2=0x7ff314151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff31b200640 1 --2- 172.21.15.92:0/706773648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff314154d50 0x7ff314151d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc033000 tx=0x7ff2fc004730 comp rx=0 tx=0).stop 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.375+0000 7ff319200640 1 -- 172.21.15.92:0/706773648 reap_dead start 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.376+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 shutdown_connections 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.376+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 >> 172.21.15.92:0/706773648 conn(0x7ff314072970 msgr2=0x7ff3140ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.376+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 shutdown_connections 2024-03-19T03:06:19.379 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.376+0000 7ff31b200640 1 -- 172.21.15.92:0/706773648 wait complete. 2024-03-19T03:06:19.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-19T03:06:19.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.585+0000 7feb53000640 1 Processor -- start 2024-03-19T03:06:19.587 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.585+0000 7feb53000640 1 -- start start 2024-03-19T03:06:19.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb53000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb53000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4c0cdd10 con 0x7feb4c0c7e60 2024-03-19T03:06:19.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb53000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4c0cd190 con 0x7feb4c0d0a40 2024-03-19T03:06:19.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb53000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c0c79d0 con 0x7feb4c0d0660 2024-03-19T03:06:19.589 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb51c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb51c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36374/0 (socket says 172.21.15.92:36374) 2024-03-19T03:06:19.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb51c00640 1 -- 172.21.15.92:0/2000055757 learned_addr learned my addr 172.21.15.92:0/2000055757 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:19.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb51200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7feb4c0d0a40 0x7feb4c0cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55848/0 (socket says 172.21.15.92:55848) 2024-03-19T03:06:19.590 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.587+0000 7feb52600640 1 --1- 172.21.15.92:0/2000055757 >> v1:172.21.15.92:6789/0 conn(0x7feb4c0c7e60 0x7feb4c1549b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53332/0 (socket says 172.21.15.92:53332) 2024-03-19T03:06:19.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.588+0000 7feb51c00640 1 -- 172.21.15.92:0/2000055757 >> v1:172.21.15.159:6789/0 conn(0x7feb4c0d0a40 legacy=0x7feb4c0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.591 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.588+0000 7feb4be00640 1 -- 172.21.15.92:0/2000055757 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316722888 0 0) 0x7feb4c0cd190 con 0x7feb4c0d0a40 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.588+0000 7feb51c00640 1 -- 172.21.15.92:0/2000055757 >> v1:172.21.15.92:6789/0 conn(0x7feb4c0c7e60 legacy=0x7feb4c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.588+0000 7feb51c00640 1 -- 172.21.15.92:0/2000055757 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c0cd470 con 0x7feb4c0d0660 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.589+0000 7feb51c00640 1 --2- 172.21.15.92:0/2000055757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb4002ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe9552f980db25a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.589+0000 7feb4be00640 1 -- 172.21.15.92:0/2000055757 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4002d850 con 0x7feb4c0d0660 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.589+0000 7feb4be00640 1 -- 172.21.15.92:0/2000055757 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4002d9f0 con 0x7feb4c0d0660 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.589+0000 7feb4be00640 1 -- 172.21.15.92:0/2000055757 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4002ddc0 con 0x7feb4c0d0660 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 -- 172.21.15.92:0/2000055757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 msgr2=0x7feb4c0cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 --2- 172.21.15.92:0/2000055757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb4002ba90 comp rx=0 tx=0).stop 2024-03-19T03:06:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 -- 172.21.15.92:0/2000055757 shutdown_connections 2024-03-19T03:06:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 --2- 172.21.15.92:0/2000055757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0660 0x7feb4c0cca00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 -- 172.21.15.92:0/2000055757 >> 172.21.15.92:0/2000055757 conn(0x7feb4c072b50 msgr2=0x7feb4c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 -- 172.21.15.92:0/2000055757 shutdown_connections 2024-03-19T03:06:19.593 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.590+0000 7feb53000640 1 -- 172.21.15.92:0/2000055757 wait complete. 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.591+0000 7feb53000640 1 Processor -- start 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 -- start start 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4c0d0660 0x7feb4c1f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0a40 0x7feb4c15a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c0cda30 con 0x7feb4c0c7e60 2024-03-19T03:06:19.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c0ccf40 con 0x7feb4c0d0660 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.592+0000 7feb53000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c0cbbc0 con 0x7feb4c0d0a40 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4c0d0660 0x7feb4c1f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47508/0 (socket says 172.21.15.92:47508) 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 -- 172.21.15.92:0/3026234217 learned_addr learned my addr 172.21.15.92:0/3026234217 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb52600640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0a40 0x7feb4c15a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 -- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0a40 msgr2=0x7feb4c15a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0a40 0x7feb4c15a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 -- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4c0d0660 msgr2=0x7feb4c1f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4c0d0660 0x7feb4c1f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 -- 172.21.15.92:0/3026234217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c1563a0 con 0x7feb4c0c7e60 2024-03-19T03:06:19.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.593+0000 7feb51c00640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.594+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40047070 con 0x7feb4c0c7e60 2024-03-19T03:06:19.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.594+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb40037ac0 con 0x7feb4c0c7e60 2024-03-19T03:06:19.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.594+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4002e420 con 0x7feb4c0c7e60 2024-03-19T03:06:19.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.594+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c15b440 con 0x7feb4c0c7e60 2024-03-19T03:06:19.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.594+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c15b700 con 0x7feb4c0c7e60 2024-03-19T03:06:19.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.595+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c04f580 con 0x7feb4c0c7e60 2024-03-19T03:06:19.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.596+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feb4004b430 con 0x7feb4c0c7e60 2024-03-19T03:06:19.599 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.597+0000 7feb49600640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 0x7feb2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:19.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.597+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7feb400ac780 con 0x7feb4c0c7e60 2024-03-19T03:06:19.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.597+0000 7feb51200640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 0x7feb2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:19.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.598+0000 7feb51200640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 0x7feb2c06e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feb38000f60 tx=0x7feb38007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:19.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.602+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feb4007a9d0 con 0x7feb4c0c7e60 2024-03-19T03:06:19.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.807+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7feb4c0cbe20 con 0x7feb2c06bbd0 2024-03-19T03:06:19.809 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.807+0000 7feb49600640 1 -- 172.21.15.92:0/3026234217 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7feb4c0cbe20 con 0x7feb2c06bbd0 2024-03-19T03:06:19.809 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 4 to deep-scrub 2024-03-19T03:06:19.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 msgr2=0x7feb2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 0x7feb2c06e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feb38000f60 tx=0x7feb38007e40 comp rx=0 tx=0).stop 2024-03-19T03:06:19.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 msgr2=0x7feb4c159560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:19.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb40004730 comp rx=0 tx=0).stop 2024-03-19T03:06:19.812 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 shutdown_connections 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7feb2c06bbd0 0x7feb2c06e090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feb4c0d0a40 0x7feb4c15a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feb4c0d0660 0x7feb4c1f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 --2- 172.21.15.92:0/3026234217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb4c0c7e60 0x7feb4c159560 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 >> 172.21.15.92:0/3026234217 conn(0x7feb4c072b50 msgr2=0x7feb4c0cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 shutdown_connections 2024-03-19T03:06:19.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:19.810+0000 7feb53000640 1 -- 172.21.15.92:0/3026234217 wait complete. 2024-03-19T03:06:19.831 INFO:tasks.ceph:Scrubbing osd.5 2024-03-19T03:06:19.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:20.045 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.043+0000 7f7cca800640 1 Processor -- start 2024-03-19T03:06:20.045 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.043+0000 7f7cca800640 1 -- start start 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.044+0000 7f7cca800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7cc40c3a80 0x7f7cc406b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cca800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 0x7f7cc4071ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cca800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc4157c70 con 0x7f7cc414e4c0 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cca800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc40c3ec0 con 0x7f7cc40c3a80 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cca800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc4158880 con 0x7f7cc406b780 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7cc414e4c0 0x7f7cc4072b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55854/0 (socket says 172.21.15.92:55854) 2024-03-19T03:06:20.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3400640 1 -- 172.21.15.92:0/4214730700 learned_addr learned my addr 172.21.15.92:0/4214730700 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/4214730700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 0x7f7cc4071ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cbbe00640 1 --2- 172.21.15.92:0/4214730700 >> v2:172.21.15.92:3300/0 conn(0x7f7cc40c3a80 0x7f7cc406b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc2a00640 1 -- 172.21.15.92:0/4214730700 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163454936 0 0) 0x7f7cc4157c70 con 0x7f7cc414e4c0 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3e00640 1 -- 172.21.15.92:0/4214730700 >> v2:172.21.15.92:3300/0 conn(0x7f7cc40c3a80 msgr2=0x7f7cc406b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/4214730700 >> v2:172.21.15.92:3300/0 conn(0x7f7cc40c3a80 0x7f7cc406b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3e00640 1 -- 172.21.15.92:0/4214730700 >> v1:172.21.15.159:6789/0 conn(0x7f7cc414e4c0 legacy=0x7f7cc4072b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.045+0000 7f7cc3e00640 1 -- 172.21.15.92:0/4214730700 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4157f50 con 0x7f7cc406b780 2024-03-19T03:06:20.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.046+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/4214730700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 0x7f7cc4071ba0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009d90 tx=0x7f7cbc02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be49c0bb3ef47914 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cc2a00640 1 -- 172.21.15.92:0/4214730700 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc039070 con 0x7f7cc406b780 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cc2a00640 1 -- 172.21.15.92:0/4214730700 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cbc02cce0 con 0x7f7cc406b780 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cc2a00640 1 -- 172.21.15.92:0/4214730700 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc034660 con 0x7f7cc406b780 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 -- 172.21.15.92:0/4214730700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 msgr2=0x7f7cc4071ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 --2- 172.21.15.92:0/4214730700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 0x7f7cc4071ba0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009d90 tx=0x7f7cbc02bf30 comp rx=0 tx=0).stop 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 -- 172.21.15.92:0/4214730700 shutdown_connections 2024-03-19T03:06:20.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 --2- 172.21.15.92:0/4214730700 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc406b780 0x7f7cc4071ba0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 --2- 172.21.15.92:0/4214730700 >> v2:172.21.15.92:3300/0 conn(0x7f7cc40c3a80 0x7f7cc406b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 -- 172.21.15.92:0/4214730700 >> 172.21.15.92:0/4214730700 conn(0x7f7cc40cc1f0 msgr2=0x7f7cc40cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 -- 172.21.15.92:0/4214730700 shutdown_connections 2024-03-19T03:06:20.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.047+0000 7f7cca800640 1 -- 172.21.15.92:0/4214730700 wait complete. 2024-03-19T03:06:20.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.049+0000 7f7cca800640 1 Processor -- start 2024-03-19T03:06:20.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.049+0000 7f7cca800640 1 -- start start 2024-03-19T03:06:20.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 0x7f7cc41e2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 0x7f7cc41e3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cc414e4c0 0x7f7cc4145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc414e8a0 con 0x7f7cc414e4c0 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc4158510 con 0x7f7cc406b780 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cca800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc414ea20 con 0x7f7cc40c3a80 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cc3400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 0x7f7cc41e2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 0x7f7cc41e3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cc3400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 0x7f7cc41e2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39730/0 (socket says 172.21.15.92:39730) 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cc3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cc414e4c0 0x7f7cc4145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 0x7f7cc41e3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36396/0 (socket says 172.21.15.92:36396) 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.050+0000 7f7cc3400640 1 -- 172.21.15.92:0/1060761053 learned_addr learned my addr 172.21.15.92:0/1060761053 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 -- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 msgr2=0x7f7cc41e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 0x7f7cc41e2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 -- 172.21.15.92:0/1060761053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cc414e4c0 msgr2=0x7f7cc4145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cc414e4c0 0x7f7cc4145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4149ef0 con 0x7f7cc40c3a80 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cc3400640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cc406b780 0x7f7cc41e2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:20.055 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cc414e4c0 0x7f7cc4145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbbe00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 0x7f7cc41e3270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e850 tx=0x7f7cac00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac017070 con 0x7f7cc40c3a80 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cac00c940 con 0x7f7cc40c3a80 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00cd10 con 0x7f7cc40c3a80 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.051+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc414a1d0 con 0x7f7cc40c3a80 2024-03-19T03:06:20.056 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.052+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc41e2250 con 0x7f7cc40c3a80 2024-03-19T03:06:20.057 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.052+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7cc4149ca0 con 0x7f7cc40c3a80 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.053+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7cac01a070 con 0x7f7cc40c3a80 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cbb400640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7ca006bc00 0x7f7ca006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f7cac08af50 con 0x7f7cc40c3a80 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cc3400640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7ca006bc00 0x7f7ca006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cbb400640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f7ca0074300 0x7f7ca00767a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ca0076e70 con 0x7f7ca0074300 2024-03-19T03:06:20.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7cac08d570 con 0x7f7cc40c3a80 2024-03-19T03:06:20.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.055+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f7ca0074300 0x7f7ca00767a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.056+0000 7f7cc3400640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7ca006bc00 0x7f7ca006e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4000f60 tx=0x7f7cb4007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.056+0000 7f7cc3e00640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f7ca0074300 0x7f7ca00767a0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.059 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.057+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== osd.5 v2:172.21.15.159:6824/1484023345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ca0076e70 con 0x7f7ca0074300 2024-03-19T03:06:20.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.071+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 --> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7cc40684c0 con 0x7f7ca0074300 2024-03-19T03:06:20.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.072+0000 7f7cbb400640 1 -- 172.21.15.92:0/1060761053 <== osd.5 v2:172.21.15.159:6824/1484023345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7cc40684c0 con 0x7f7ca0074300 2024-03-19T03:06:20.074 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:20.074 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:20.074 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:20.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f7ca0074300 msgr2=0x7f7ca00767a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6824/1484023345,v1:172.21.15.159:6825/1484023345] conn(0x7f7ca0074300 0x7f7ca00767a0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7ca006bc00 msgr2=0x7f7ca006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7ca006bc00 0x7f7ca006e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4000f60 tx=0x7f7cb4007ea0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 msgr2=0x7f7cc41e3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 --2- 172.21.15.92:0/1060761053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cc40c3a80 0x7f7cc41e3270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e850 tx=0x7f7cac00ed10 comp rx=0 tx=0).stop 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cc3e00640 1 -- 172.21.15.92:0/1060761053 reap_dead start 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 shutdown_connections 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 >> 172.21.15.92:0/1060761053 conn(0x7f7cc40cc1f0 msgr2=0x7f7cc406ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 shutdown_connections 2024-03-19T03:06:20.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.073+0000 7f7cca800640 1 -- 172.21.15.92:0/1060761053 wait complete. 2024-03-19T03:06:20.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.293+0000 7f7bbfc00640 1 Processor -- start 2024-03-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.294+0000 7f7bbfc00640 1 -- start start 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbfc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bb8151ce0 0x7f7bb8158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbfc00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb80c7da0 0x7f7bb80c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbfc00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bb80c3b80 con 0x7f7bb8151900 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbfc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb8158860 con 0x7f7bb8151ce0 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbfc00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb8072e70 con 0x7f7bb80c7da0 2024-03-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbdc00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7bb80c7da0 0x7f7bb80c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbd200640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7bb8151900 0x7f7bb80723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55858/0 (socket says 172.21.15.92:55858) 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.295+0000 7f7bbd200640 1 -- 172.21.15.92:0/2384345602 learned_addr learned my addr 172.21.15.92:0/2384345602 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bafe00640 1 --2- 172.21.15.92:0/2384345602 >> v2:172.21.15.92:3300/0 conn(0x7f7bb8151ce0 0x7f7bb8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bbdc00640 1 -- 172.21.15.92:0/2384345602 >> v2:172.21.15.92:3300/0 conn(0x7f7bb8151ce0 msgr2=0x7f7bb8158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bbdc00640 1 --2- 172.21.15.92:0/2384345602 >> v2:172.21.15.92:3300/0 conn(0x7f7bb8151ce0 0x7f7bb8158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bb7e00640 1 -- 172.21.15.92:0/2384345602 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551330232 0 0) 0x7f7bb80c3b80 con 0x7f7bb8151900 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bbdc00640 1 -- 172.21.15.92:0/2384345602 >> v1:172.21.15.159:6789/0 conn(0x7f7bb8151900 legacy=0x7f7bb80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.296+0000 7f7bbdc00640 1 -- 172.21.15.92:0/2384345602 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb8073150 con 0x7f7bb80c7da0 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.297+0000 7f7bbdc00640 1 --2- 172.21.15.92:0/2384345602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb80c81f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8073700 tx=0x7f7bb002e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=336ec5e10d480e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bb7e00640 1 -- 172.21.15.92:0/2384345602 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb002eb00 con 0x7f7bb80c7da0 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bb7e00640 1 -- 172.21.15.92:0/2384345602 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bb002eca0 con 0x7f7bb80c7da0 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bb7e00640 1 -- 172.21.15.92:0/2384345602 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0009760 con 0x7f7bb80c7da0 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 -- 172.21.15.92:0/2384345602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 msgr2=0x7f7bb80c81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/2384345602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb80c81f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8073700 tx=0x7f7bb002e160 comp rx=0 tx=0).stop 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 -- 172.21.15.92:0/2384345602 shutdown_connections 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/2384345602 >> v2:172.21.15.92:3300/0 conn(0x7f7bb8151ce0 0x7f7bb8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/2384345602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb80c81f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 -- 172.21.15.92:0/2384345602 >> 172.21.15.92:0/2384345602 conn(0x7f7bb80c8d90 msgr2=0x7f7bb80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 -- 172.21.15.92:0/2384345602 shutdown_connections 2024-03-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.298+0000 7f7bbfc00640 1 -- 172.21.15.92:0/2384345602 wait complete. 2024-03-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.300+0000 7f7bbfc00640 1 Processor -- start 2024-03-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.300+0000 7f7bbfc00640 1 -- start start 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb8151900 0x7f7bb81f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb8151ce0 0x7f7bb8144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb80c9af0 con 0x7f7bb8151ce0 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8071eb0 con 0x7f7bb80c7da0 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbfc00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb80c9c70 con 0x7f7bb8151900 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.301+0000 7f7bbd200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39754/0 (socket says 172.21.15.92:39754) 2024-03-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 -- 172.21.15.92:0/622530880 learned_addr learned my addr 172.21.15.92:0/622530880 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bafe00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb8151900 0x7f7bb81f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbdc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb8151ce0 0x7f7bb8144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 -- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb8151900 msgr2=0x7f7bb81f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb8151900 0x7f7bb81f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 -- 172.21.15.92:0/622530880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb8151ce0 msgr2=0x7f7bb8144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb8151ce0 0x7f7bb8144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 -- 172.21.15.92:0/622530880 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb806f690 con 0x7f7bb80c7da0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.302+0000 7f7bbd200640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c4d0 tx=0x7f7ba800c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.303+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8018070 con 0x7f7bb80c7da0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.303+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba8007500 con 0x7f7bb80c7da0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.303+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8004a20 con 0x7f7bb80c7da0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.303+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb81559c0 con 0x7f7bb80c7da0 2024-03-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.303+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb8152460 con 0x7f7bb80c7da0 2024-03-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.304+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb80684c0 con 0x7f7bb80c7da0 2024-03-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.305+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ba801b070 con 0x7f7bb80c7da0 2024-03-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.306+0000 7f7bb5600640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 0x7f7b9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.306+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f7ba8089f90 con 0x7f7bb80c7da0 2024-03-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.306+0000 7f7bafe00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 0x7f7b9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.307+0000 7f7bafe00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 0x7f7b9406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7bb815bd50 tx=0x7f7ba0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.312+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ba8058190 con 0x7f7bb80c7da0 2024-03-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.531+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f7bb8152210 con 0x7f7b9406bbd0 2024-03-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.532+0000 7f7bb5600640 1 -- 172.21.15.92:0/622530880 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7bb8152210 con 0x7f7b9406bbd0 2024-03-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 5 to deep-scrub 2024-03-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 msgr2=0x7f7b9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 0x7f7b9406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7bb815bd50 tx=0x7f7ba0009190 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 msgr2=0x7f7bb8156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c4d0 tx=0x7f7ba800c990 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 shutdown_connections 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7b9406bbd0 0x7f7b9406e090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb8151ce0 0x7f7bb8144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7bb8151900 0x7f7bb81f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.534+0000 7f7bbfc00640 1 --2- 172.21.15.92:0/622530880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bb80c7da0 0x7f7bb8156180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.535+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 >> 172.21.15.92:0/622530880 conn(0x7f7bb80c8d90 msgr2=0x7f7bb80c8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.535+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 shutdown_connections 2024-03-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.535+0000 7f7bbfc00640 1 -- 172.21.15.92:0/622530880 wait complete. 2024-03-19T03:06:20.555 INFO:tasks.ceph:Scrubbing osd.6 2024-03-19T03:06:20.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.778+0000 7f4c38a00640 1 Processor -- start 2024-03-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.778+0000 7f4c38a00640 1 -- start start 2024-03-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.779+0000 7f4c38a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4c340c3a80 0x7f4c3406b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.779+0000 7f4c38a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c34157c70 con 0x7f4c3406b780 2024-03-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.779+0000 7f4c38a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c34157f50 con 0x7f4c3414e4c0 2024-03-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.779+0000 7f4c38a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4c340c3ec0 con 0x7f4c340c3a80 2024-03-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4c3414e4c0 0x7f4c34072b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55860/0 (socket says 172.21.15.92:55860) 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31e00640 1 -- 172.21.15.92:0/4000833819 learned_addr learned my addr 172.21.15.92:0/4000833819 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c2be00640 1 --2- 172.21.15.92:0/4000833819 >> v2:172.21.15.92:3300/0 conn(0x7f4c340c3a80 0x7f4c3406b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 271153303 0 0) 0x7f4c34157f50 con 0x7f4c3414e4c0 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c18003520 con 0x7f4c3414e4c0 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287519881 0 0) 0x7f4c34157c70 con 0x7f4c3406b780 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c34157f50 con 0x7f4c3406b780 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c2be00640 1 -- 172.21.15.92:0/4000833819 >> v1:172.21.15.159:6789/0 conn(0x7f4c3414e4c0 legacy=0x7f4c34072b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c2be00640 1 -- 172.21.15.92:0/4000833819 >> v1:172.21.15.92:6789/0 conn(0x7f4c3406b780 legacy=0x7f4c34071ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.780+0000 7f4c2be00640 1 -- 172.21.15.92:0/4000833819 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c34158230 con 0x7f4c340c3a80 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c2be00640 1 --2- 172.21.15.92:0/4000833819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c340c3a80 0x7f4c3406b240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3243322eea1b4571 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c02f810 con 0x7f4c340c3a80 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c02f9b0 con 0x7f4c340c3a80 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c31400640 1 -- 172.21.15.92:0/4000833819 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c02fd80 con 0x7f4c340c3a80 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c38a00640 1 -- 172.21.15.92:0/4000833819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c340c3a80 msgr2=0x7f4c3406b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.782+0000 7f4c38a00640 1 --2- 172.21.15.92:0/4000833819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c340c3a80 0x7f4c3406b240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b9f0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.783+0000 7f4c38a00640 1 -- 172.21.15.92:0/4000833819 shutdown_connections 2024-03-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.783+0000 7f4c38a00640 1 --2- 172.21.15.92:0/4000833819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c340c3a80 0x7f4c3406b240 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.783+0000 7f4c38a00640 1 -- 172.21.15.92:0/4000833819 >> 172.21.15.92:0/4000833819 conn(0x7f4c340cc1f0 msgr2=0x7f4c340cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.783+0000 7f4c38a00640 1 -- 172.21.15.92:0/4000833819 shutdown_connections 2024-03-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.783+0000 7f4c38a00640 1 -- 172.21.15.92:0/4000833819 wait complete. 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.784+0000 7f4c38a00640 1 Processor -- start 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.784+0000 7f4c38a00640 1 -- start start 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c3406b780 0x7f4c34149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 0x7f4c340cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c3414e4c0 0x7f4c3414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3414e8a0 con 0x7f4c3406b780 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4c34158880 con 0x7f4c340c3a80 2024-03-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.785+0000 7f4c38a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3414ea20 con 0x7f4c3414e4c0 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 0x7f4c340cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 0x7f4c340cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39770/0 (socket says 172.21.15.92:39770) 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c31e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c3406b780 0x7f4c34149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 -- 172.21.15.92:0/3312754900 learned_addr learned my addr 172.21.15.92:0/3312754900 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c32800640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c3414e4c0 0x7f4c3414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 -- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c3414e4c0 msgr2=0x7f4c3414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c3414e4c0 0x7f4c3414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 -- 172.21.15.92:0/3312754900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c3406b780 msgr2=0x7f4c34149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c3406b780 0x7f4c34149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c34156e20 con 0x7f4c340c3a80 2024-03-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.786+0000 7f4c2be00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 0x7f4c340cb3e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c02c000 tx=0x7f4c1c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.787+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c047070 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.787+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c0378f0 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.787+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c340c6d50 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.787+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c037ce0 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.787+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c3414a9f0 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.788+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c341578c0 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.789+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4c1c04a070 con 0x7f4c340c3a80 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c2a000640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f4c1c0ac770 con 0x7f4c340c3a80 2024-03-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c31e00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c2a000640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7f4c100742e0 0x7f4c10076780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c10076e50 con 0x7f4c100742e0 2024-03-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4c1c0b3050 con 0x7f4c340c3a80 2024-03-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c32800640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7f4c100742e0 0x7f4c10076780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.790+0000 7f4c31e00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000f60 tx=0x7f4c24007d70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.791+0000 7f4c32800640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7f4c100742e0 0x7f4c10076780 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.792+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== osd.6 v2:172.21.15.159:6800/2808929873 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c10076e50 con 0x7f4c100742e0 2024-03-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.806+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 --> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4c340684c0 con 0x7f4c100742e0 2024-03-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.806+0000 7f4c2a000640 1 -- 172.21.15.92:0/3312754900 <== osd.6 v2:172.21.15.159:6800/2808929873 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4c340684c0 con 0x7f4c100742e0 2024-03-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7f4c100742e0 msgr2=0x7f4c10076780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6800/2808929873,v1:172.21.15.159:6801/2808929873] conn(0x7f4c100742e0 0x7f4c10076780 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c1006bbd0 msgr2=0x7f4c1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000f60 tx=0x7f4c24007d70 comp rx=0 tx=0).stop 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 msgr2=0x7f4c340cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 --2- 172.21.15.92:0/3312754900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c340c3a80 0x7f4c340cb3e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c02c000 tx=0x7f4c1c004730 comp rx=0 tx=0).stop 2024-03-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c32800640 1 -- 172.21.15.92:0/3312754900 reap_dead start 2024-03-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 shutdown_connections 2024-03-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 >> 172.21.15.92:0/3312754900 conn(0x7f4c340cc1f0 msgr2=0x7f4c340c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 shutdown_connections 2024-03-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:20.807+0000 7f4c38a00640 1 -- 172.21.15.92:0/3312754900 wait complete. 2024-03-19T03:06:20.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-19T03:06:21.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.024+0000 7f02fe000640 1 Processor -- start 2024-03-19T03:06:21.026 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.024+0000 7f02fe000640 1 -- start start 2024-03-19T03:06:21.027 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02fe000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 0x7f02f806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02fe000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02f81593d0 0x7f02f80c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02fe000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f80c9a80 con 0x7f02f80c6470 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02fe000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f02f80c6c90 con 0x7f02f81593d0 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02fe000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f02f8159910 con 0x7f02f8152bf0 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 0x7f02f806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02f81593d0 0x7f02f80c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f6a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f02f80c6470 0x7f02f80c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:53354/0 (socket says 172.21.15.92:53354) 2024-03-19T03:06:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f6a00640 1 -- 172.21.15.92:0/2344069657 learned_addr learned my addr 172.21.15.92:0/2344069657 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7400640 1 -- 172.21.15.92:0/2344069657 >> v2:172.21.15.92:3300/0 conn(0x7f02f81593d0 msgr2=0x7f02f80c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7400640 1 --2- 172.21.15.92:0/2344069657 >> v2:172.21.15.92:3300/0 conn(0x7f02f81593d0 0x7f02f80c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.030 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7400640 1 -- 172.21.15.92:0/2344069657 >> v1:172.21.15.92:6789/0 conn(0x7f02f80c6470 legacy=0x7f02f80c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.026+0000 7f02f7400640 1 -- 172.21.15.92:0/2344069657 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f80c8650 con 0x7f02f8152bf0 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02f7400640 1 --2- 172.21.15.92:0/2344069657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 0x7f02f806eef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02e80097a0 tx=0x7f02e802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3493dd6edd64b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02f6000640 1 -- 172.21.15.92:0/2344069657 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8031780 con 0x7f02f8152bf0 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02f6000640 1 -- 172.21.15.92:0/2344069657 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e8037070 con 0x7f02f8152bf0 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02f6000640 1 -- 172.21.15.92:0/2344069657 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8031b40 con 0x7f02f8152bf0 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 -- 172.21.15.92:0/2344069657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 msgr2=0x7f02f806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 --2- 172.21.15.92:0/2344069657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 0x7f02f806eef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02e80097a0 tx=0x7f02e802b820 comp rx=0 tx=0).stop 2024-03-19T03:06:21.031 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 -- 172.21.15.92:0/2344069657 shutdown_connections 2024-03-19T03:06:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 --2- 172.21.15.92:0/2344069657 >> v2:172.21.15.92:3300/0 conn(0x7f02f81593d0 0x7f02f80c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 --2- 172.21.15.92:0/2344069657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f8152bf0 0x7f02f806eef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.028+0000 7f02fe000640 1 -- 172.21.15.92:0/2344069657 >> 172.21.15.92:0/2344069657 conn(0x7f02f80c8d90 msgr2=0x7f02f80c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.029+0000 7f02fe000640 1 -- 172.21.15.92:0/2344069657 shutdown_connections 2024-03-19T03:06:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.029+0000 7f02fe000640 1 -- 172.21.15.92:0/2344069657 wait complete. 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.030+0000 7f02fe000640 1 Processor -- start 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.030+0000 7f02fe000640 1 -- start start 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02f8152bf0 0x7f02f815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02f81593d0 0x7f02f8146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02f806ea60 con 0x7f02f8152bf0 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f02f806e4a0 con 0x7f02f81593d0 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02fe000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f02f806e6f0 con 0x7f02f80c6470 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02f7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.031+0000 7f02f6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02f8152bf0 0x7f02f815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:36434/0 (socket says 172.21.15.92:36434) 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 -- 172.21.15.92:0/1818084391 learned_addr learned my addr 172.21.15.92:0/1818084391 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7e00640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02f81593d0 0x7f02f8146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 -- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02f81593d0 msgr2=0x7f02f8146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02f81593d0 0x7f02f8146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 -- 172.21.15.92:0/1818084391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02f8152bf0 msgr2=0x7f02f815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02f8152bf0 0x7f02f815a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 -- 172.21.15.92:0/1818084391 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f814f600 con 0x7f02f80c6470 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02f7400640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098c0 tx=0x7f02e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.032+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8032610 con 0x7f02f80c6470 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.033+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e80327b0 con 0x7f02f80c6470 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.033+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f8149a70 con 0x7f02f80c6470 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.033+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8032ba0 con 0x7f02f80c6470 2024-03-19T03:06:21.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.033+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f814a030 con 0x7f02f80c6470 2024-03-19T03:06:21.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.034+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8151bc0 con 0x7f02f80c6470 2024-03-19T03:06:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.035+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02e804e440 con 0x7f02f80c6470 2024-03-19T03:06:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.035+0000 7f02ef400640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 0x7f02d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.035+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f02e80ac930 con 0x7f02f80c6470 2024-03-19T03:06:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.036+0000 7f02f6a00640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 0x7f02d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.036+0000 7f02f6a00640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 0x7f02d406e0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f02f815ce20 tx=0x7f02e0007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.040+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02f8151bc0 con 0x7f02f80c6470 2024-03-19T03:06:21.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.242+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f02f80684c0 con 0x7f02d406bc00 2024-03-19T03:06:21.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.243+0000 7f02ef400640 1 -- 172.21.15.92:0/1818084391 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f02f80684c0 con 0x7f02d406bc00 2024-03-19T03:06:21.245 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 6 to deep-scrub 2024-03-19T03:06:21.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 msgr2=0x7f02d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 0x7f02d406e0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f02f815ce20 tx=0x7f02e0007e80 comp rx=0 tx=0).stop 2024-03-19T03:06:21.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 msgr2=0x7f02f814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098c0 tx=0x7f02e80046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 shutdown_connections 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f02d406bc00 0x7f02d406e0c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02f81593d0 0x7f02f8146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02f8152bf0 0x7f02f815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 --2- 172.21.15.92:0/1818084391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02f80c6470 0x7f02f814eea0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.245+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 >> 172.21.15.92:0/1818084391 conn(0x7f02f80c8d90 msgr2=0x7f02f8148f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.246+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 shutdown_connections 2024-03-19T03:06:21.249 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.246+0000 7f02fe000640 1 -- 172.21.15.92:0/1818084391 wait complete. 2024-03-19T03:06:21.266 INFO:tasks.ceph:Scrubbing osd.7 2024-03-19T03:06:21.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.459+0000 7f08a7000640 1 Processor -- start 2024-03-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.459+0000 7f08a7000640 1 -- start start 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f08a7000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f08a00c36f0 0x7f08a006b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f08a7000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a00c4000 con 0x7f08a006b780 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f08a7000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a00c42c0 con 0x7f08a0152d70 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f08a7000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f08a006bbb0 con 0x7f08a00c36f0 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f089fe00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f08a0152d70 0x7f08a00c8160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55876/0 (socket says 172.21.15.92:55876) 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f089fe00640 1 -- 172.21.15.92:0/2686738498 learned_addr learned my addr 172.21.15.92:0/2686738498 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f089f400640 1 --2- 172.21.15.92:0/2686738498 >> v2:172.21.15.159:3300/0 conn(0x7f08a00c36f0 0x7f08a006b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.461+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355867885 0 0) 0x7f08a00c42c0 con 0x7f08a0152d70 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.462+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0884003520 con 0x7f08a0152d70 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.462+0000 7f089f400640 1 -- 172.21.15.92:0/2686738498 >> v1:172.21.15.159:6789/0 conn(0x7f08a0152d70 legacy=0x7f08a00c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.462+0000 7f089f400640 1 -- 172.21.15.92:0/2686738498 >> v1:172.21.15.92:6789/0 conn(0x7f08a006b780 legacy=0x7f08a0158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.462+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136118796 0 0) 0x7f08a00c4000 con 0x7f08a006b780 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.462+0000 7f089f400640 1 -- 172.21.15.92:0/2686738498 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a00c45a0 con 0x7f08a00c36f0 2024-03-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f089f400640 1 --2- 172.21.15.92:0/2686738498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a00c36f0 0x7f08a006b240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097a0 tx=0x7f088c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46faf07cedd7d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c034070 con 0x7f08a00c36f0 2024-03-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f088c030610 con 0x7f08a00c36f0 2024-03-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f089ea00640 1 -- 172.21.15.92:0/2686738498 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c0309e0 con 0x7f08a00c36f0 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f08a7000640 1 -- 172.21.15.92:0/2686738498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a00c36f0 msgr2=0x7f08a006b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.463+0000 7f08a7000640 1 --2- 172.21.15.92:0/2686738498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a00c36f0 0x7f08a006b240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097a0 tx=0x7f088c02b960 comp rx=0 tx=0).stop 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.464+0000 7f08a7000640 1 -- 172.21.15.92:0/2686738498 shutdown_connections 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.464+0000 7f08a7000640 1 --2- 172.21.15.92:0/2686738498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a00c36f0 0x7f08a006b240 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.464+0000 7f08a7000640 1 -- 172.21.15.92:0/2686738498 >> 172.21.15.92:0/2686738498 conn(0x7f08a00cc1f0 msgr2=0x7f08a00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.464+0000 7f08a7000640 1 -- 172.21.15.92:0/2686738498 shutdown_connections 2024-03-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.464+0000 7f08a7000640 1 -- 172.21.15.92:0/2686738498 wait complete. 2024-03-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.465+0000 7f08a7000640 1 Processor -- start 2024-03-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.465+0000 7f08a7000640 1 -- start start 2024-03-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 0x7f08a014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a00c36f0 0x7f08a014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08a0152d70 0x7f08a014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08a00c4b60 con 0x7f08a0152d70 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f08a00c4ed0 con 0x7f08a006b780 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.466+0000 7f08a7000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f08a0153150 con 0x7f08a00c36f0 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 0x7f08a014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089f400640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a00c36f0 0x7f08a014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 0x7f08a014f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:39798/0 (socket says 172.21.15.92:39798) 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 -- 172.21.15.92:0/313215612 learned_addr learned my addr 172.21.15.92:0/313215612 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f08a5000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08a0152d70 0x7f08a014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 -- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a00c36f0 msgr2=0x7f08a014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a00c36f0 0x7f08a014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 -- 172.21.15.92:0/313215612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08a0152d70 msgr2=0x7f08a014e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08a0152d70 0x7f08a014e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a014d190 con 0x7f08a006b780 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f089fe00640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 0x7f08a014f730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f089000c440 tx=0x7f089000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890017070 con 0x7f08a006b780 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0890005ce0 con 0x7f08a006b780 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.467+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08900072b0 con 0x7f08a006b780 2024-03-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.468+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a014d470 con 0x7f08a006b780 2024-03-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.468+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a014da30 con 0x7f08a006b780 2024-03-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.468+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f08a00684c0 con 0x7f08a006b780 2024-03-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.470+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f089001a070 con 0x7f08a006b780 2024-03-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.470+0000 7f0897e00640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f088006bbd0 0x7f088006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.470+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f0890089fe0 con 0x7f08a006b780 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f089f400640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f088006bbd0 0x7f088006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f0897e00640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7f08800742d0 0x7f0880076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0890016ec0 con 0x7f08800742d0 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f089008c620 con 0x7f08a006b780 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f08a5000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7f08800742d0 0x7f0880076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f089f400640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f088006bbd0 0x7f088006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f088c036000 tx=0x7f088c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.471+0000 7f08a5000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7f08800742d0 0x7f0880076770 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.472+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== osd.7 v2:172.21.15.159:6816/3107756136 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0890016ec0 con 0x7f08800742d0 2024-03-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.486+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 --> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f08a00c7730 con 0x7f08800742d0 2024-03-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.487+0000 7f0897e00640 1 -- 172.21.15.92:0/313215612 <== osd.7 v2:172.21.15.159:6816/3107756136 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f08a00c7730 con 0x7f08800742d0 2024-03-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.487+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7f08800742d0 msgr2=0x7f0880076770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6816/3107756136,v1:172.21.15.159:6817/3107756136] conn(0x7f08800742d0 0x7f0880076770 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f088006bbd0 msgr2=0x7f088006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f088006bbd0 0x7f088006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f088c036000 tx=0x7f088c008030 comp rx=0 tx=0).stop 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 msgr2=0x7f08a014f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 --2- 172.21.15.92:0/313215612 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a006b780 0x7f08a014f730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f089000c440 tx=0x7f089000c900 comp rx=0 tx=0).stop 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a5000640 1 -- 172.21.15.92:0/313215612 reap_dead start 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 shutdown_connections 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 >> 172.21.15.92:0/313215612 conn(0x7f08a00cc1f0 msgr2=0x7f08a01582e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 shutdown_connections 2024-03-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.488+0000 7f08a7000640 1 -- 172.21.15.92:0/313215612 wait complete. 2024-03-19T03:06:21.506 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-19T03:06:21.713 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.712+0000 7fd4d5800640 1 Processor -- start 2024-03-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.712+0000 7fd4d5800640 1 -- start start 2024-03-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4d5800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 0x7fd4d006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4d5800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4d01593d0 0x7fd4d00c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4d5800640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d00c9a80 con 0x7fd4d00c6470 2024-03-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4d5800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d00c6c90 con 0x7fd4d01593d0 2024-03-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4d5800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d0159910 con 0x7fd4d0152bf0 2024-03-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4cf400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4d01593d0 0x7fd4d00c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4cea00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 0x7fd4d006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.713+0000 7fd4c7e00640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd4d00c6470 0x7fd4d00c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55892/0 (socket says 172.21.15.92:55892) 2024-03-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4c7e00640 1 -- 172.21.15.92:0/1819435443 learned_addr learned my addr 172.21.15.92:0/1819435443 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4cf400640 1 -- 172.21.15.92:0/1819435443 >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 msgr2=0x7fd4d006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4cf400640 1 --2- 172.21.15.92:0/1819435443 >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 0x7fd4d006eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4cf400640 1 -- 172.21.15.92:0/1819435443 >> v1:172.21.15.159:6789/0 conn(0x7fd4d00c6470 legacy=0x7fd4d00c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4cf400640 1 -- 172.21.15.92:0/1819435443 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d00c8650 con 0x7fd4d01593d0 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.714+0000 7fd4cea00640 1 --2- 172.21.15.92:0/1819435443 >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 0x7fd4d006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.715+0000 7fd4cf400640 1 --2- 172.21.15.92:0/1819435443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d00c4690 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0075180 tx=0x7fd4c802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ca13f1729fa992 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.715+0000 7fd4ce000640 1 -- 172.21.15.92:0/1819435443 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8039070 con 0x7fd4d01593d0 2024-03-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.715+0000 7fd4ce000640 1 -- 172.21.15.92:0/1819435443 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c802cbe0 con 0x7fd4d01593d0 2024-03-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.715+0000 7fd4ce000640 1 -- 172.21.15.92:0/1819435443 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8034440 con 0x7fd4d01593d0 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 -- 172.21.15.92:0/1819435443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 msgr2=0x7fd4d00c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1819435443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d00c4690 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0075180 tx=0x7fd4c802be30 comp rx=0 tx=0).stop 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 -- 172.21.15.92:0/1819435443 shutdown_connections 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1819435443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d00c4690 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1819435443 >> v2:172.21.15.159:3300/0 conn(0x7fd4d0152bf0 0x7fd4d006eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 -- 172.21.15.92:0/1819435443 >> 172.21.15.92:0/1819435443 conn(0x7fd4d00c8d90 msgr2=0x7fd4d00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 -- 172.21.15.92:0/1819435443 shutdown_connections 2024-03-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.716+0000 7fd4d5800640 1 -- 172.21.15.92:0/1819435443 wait complete. 2024-03-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.718+0000 7fd4d5800640 1 Processor -- start 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.718+0000 7fd4d5800640 1 -- start start 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4d00c6470 0x7fd4d014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4d0152bf0 0x7fd4d0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d006ea60 con 0x7fd4d01593d0 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d006e4a0 con 0x7fd4d0152bf0 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4d5800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d006e6f0 con 0x7fd4d00c6470 2024-03-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4cf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4c7e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4d0152bf0 0x7fd4d0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4cea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4d00c6470 0x7fd4d014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4cf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47586/0 (socket says 172.21.15.92:47586) 2024-03-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.719+0000 7fd4cf400640 1 -- 172.21.15.92:0/1834513780 learned_addr learned my addr 172.21.15.92:0/1834513780 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 -- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4d00c6470 msgr2=0x7fd4d014fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4d00c6470 0x7fd4d014fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 -- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4d0152bf0 msgr2=0x7fd4d0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4d0152bf0 0x7fd4d0159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 -- 172.21.15.92:0/1834513780 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d014ff90 con 0x7fd4d01593d0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4cf400640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009e80 tx=0x7fd4c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8039050 con 0x7fd4d01593d0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c8045070 con 0x7fd4d01593d0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d015cb80 con 0x7fd4d01593d0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8009450 con 0x7fd4d01593d0 2024-03-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.720+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d01f0a10 con 0x7fd4d01593d0 2024-03-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.722+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d00684c0 con 0x7fd4d01593d0 2024-03-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.722+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd4c803e850 con 0x7fd4d01593d0 2024-03-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.723+0000 7fd4c6a00640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 0x7fd4ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.723+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fd4c80adaa0 con 0x7fd4d01593d0 2024-03-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.724+0000 7fd4cea00640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 0x7fd4ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.724+0000 7fd4cea00640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 0x7fd4ac06e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009560 tx=0x7fd4c0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.728+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd4c807bd90 con 0x7fd4d01593d0 2024-03-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.925+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd4d014d1f0 con 0x7fd4ac06bbd0 2024-03-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.926+0000 7fd4c6a00640 1 -- 172.21.15.92:0/1834513780 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd4d014d1f0 con 0x7fd4ac06bbd0 2024-03-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi092.stderr:instructed osd(s) 7 to deep-scrub 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 msgr2=0x7fd4ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 0x7fd4ac06e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009560 tx=0x7fd4c0009190 comp rx=0 tx=0).stop 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 msgr2=0x7fd4d015a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009e80 tx=0x7fd4c8004a40 comp rx=0 tx=0).stop 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 shutdown_connections 2024-03-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fd4ac06bbd0 0x7fd4ac06e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d01593d0 0x7fd4d015a2c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4d0152bf0 0x7fd4d0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 --2- 172.21.15.92:0/1834513780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4d00c6470 0x7fd4d014fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 >> 172.21.15.92:0/1834513780 conn(0x7fd4d00c8d90 msgr2=0x7fd4d0148fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 shutdown_connections 2024-03-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:21.929+0000 7fd4d5800640 1 -- 172.21.15.92:0/1834513780 wait complete. 2024-03-19T03:06:21.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.155+0000 7fa5d5a00640 1 Processor -- start 2024-03-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.155+0000 7fa5d5a00640 1 -- start start 2024-03-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.157+0000 7fa5d5a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa5d00c4800 0x7fa5d00c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.157+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0154940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.157+0000 7fa5d5a00640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d00c5980 con 0x7fa5d00c4be0 2024-03-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.157+0000 7fa5d5a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d00c85a0 con 0x7fa5d00c4800 2024-03-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.157+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d00c5c40 con 0x7fa5d00c61a0 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5d5000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5cfe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa5d00c4800 0x7fa5d00c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5cf400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa5d00c4be0 0x7fa5d015b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:55906/0 (socket says 172.21.15.92:55906) 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5cf400640 1 -- 172.21.15.92:0/3707222381 learned_addr learned my addr 172.21.15.92:0/3707222381 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5d5000640 1 -- 172.21.15.92:0/3707222381 >> v2:172.21.15.92:3300/0 conn(0x7fa5d00c4800 msgr2=0x7fa5d00c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5d5000640 1 --2- 172.21.15.92:0/3707222381 >> v2:172.21.15.92:3300/0 conn(0x7fa5d00c4800 0x7fa5d00c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5d5000640 1 -- 172.21.15.92:0/3707222381 >> v1:172.21.15.159:6789/0 conn(0x7fa5d00c4be0 legacy=0x7fa5d015b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5d5000640 1 -- 172.21.15.92:0/3707222381 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d00c7990 con 0x7fa5d00c61a0 2024-03-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.158+0000 7fa5cea00640 1 -- 172.21.15.92:0/3707222381 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810343887 0 0) 0x7fa5d00c5980 con 0x7fa5d00c4be0 2024-03-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.160+0000 7fa5d5000640 1 --2- 172.21.15.92:0/3707222381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0154940 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009d90 tx=0x7fa5c802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd58a74d1944306 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.160+0000 7fa5cea00640 1 -- 172.21.15.92:0/3707222381 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8039070 con 0x7fa5d00c61a0 2024-03-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.160+0000 7fa5cea00640 1 -- 172.21.15.92:0/3707222381 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c802cce0 con 0x7fa5d00c61a0 2024-03-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.160+0000 7fa5cea00640 1 -- 172.21.15.92:0/3707222381 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8034660 con 0x7fa5d00c61a0 2024-03-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 -- 172.21.15.92:0/3707222381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 msgr2=0x7fa5d0154940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/3707222381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0154940 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009d90 tx=0x7fa5c802bf30 comp rx=0 tx=0).stop 2024-03-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 -- 172.21.15.92:0/3707222381 shutdown_connections 2024-03-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/3707222381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0154940 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/3707222381 >> v2:172.21.15.92:3300/0 conn(0x7fa5d00c4800 0x7fa5d00c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 -- 172.21.15.92:0/3707222381 >> 172.21.15.92:0/3707222381 conn(0x7fa5d006fb00 msgr2=0x7fa5d006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 -- 172.21.15.92:0/3707222381 shutdown_connections 2024-03-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.161+0000 7fa5d5a00640 1 -- 172.21.15.92:0/3707222381 wait complete. 2024-03-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.163+0000 7fa5d5a00640 1 Processor -- start 2024-03-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.164+0000 7fa5d5a00640 1 -- start start 2024-03-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 0x7fa5d01f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d00c8230 con 0x7fa5d00c4800 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d00c7f50 con 0x7fa5d00c4be0 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d00c8a30 con 0x7fa5d00c61a0 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.165+0000 7fa5cf400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 0x7fa5d01f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47610/0 (socket says 172.21.15.92:47610) 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 -- 172.21.15.92:0/714005420 learned_addr learned my addr 172.21.15.92:0/714005420 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5d5000640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 -- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 msgr2=0x7fa5d0150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 -- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 msgr2=0x7fa5d01f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 0x7fa5d01f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cfe00640 1 -- 172.21.15.92:0/714005420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d006c470 con 0x7fa5d00c4800 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.166+0000 7fa5cf400640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 0x7fa5d01f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5cfe00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e9e0 tx=0x7fa5c000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0017070 con 0x7fa5d00c4800 2024-03-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c000c9d0 con 0x7fa5d00c4800 2024-03-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d0150e90 con 0x7fa5d00c4800 2024-03-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000cdc0 con 0x7fa5d00c4800 2024-03-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.167+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d006d420 con 0x7fa5d00c4800 2024-03-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.169+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d004f580 con 0x7fa5d00c4800 2024-03-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.170+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa5c001a070 con 0x7fa5d00c4800 2024-03-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.171+0000 7fa5c7e00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 0x7fa5b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.171+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fa5c008acd0 con 0x7fa5d00c4800 2024-03-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.171+0000 7fa5cf400640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 0x7fa5b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.172+0000 7fa5cf400640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 0x7fa5b006e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5d01596d0 tx=0x7fa5bc007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.178+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa5c0058eb0 con 0x7fa5d00c4800 2024-03-19T03:06:22.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.397+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5d0159700 con 0x7fa5b006bbd0 2024-03-19T03:06:22.406 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.405+0000 7fa5c7e00640 1 -- 172.21.15.92:0/714005420 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254688 (secure 0 0 0) 0x7fa5d0159700 con 0x7fa5b006bbd0 2024-03-19T03:06:22.410 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:22.411 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T03:06:22.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.409+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 msgr2=0x7fa5b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.409+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 0x7fa5b006e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5d01596d0 tx=0x7fa5bc007d50 comp rx=0 tx=0).stop 2024-03-19T03:06:22.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 msgr2=0x7fa5d01510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e9e0 tx=0x7fa5c000eea0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 shutdown_connections 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fa5b006bbd0 0x7fa5b006e090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5d00c61a0 0x7fa5d0150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5d00c4be0 0x7fa5d01f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 --2- 172.21.15.92:0/714005420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5d00c4800 0x7fa5d01510e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 >> 172.21.15.92:0/714005420 conn(0x7fa5d006fb00 msgr2=0x7fa5d015a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:22.414 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 shutdown_connections 2024-03-19T03:06:22.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:22.410+0000 7fa5d5a00640 1 -- 172.21.15.92:0/714005420 wait complete. 2024-03-19T03:06:22.433 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-03-19T03:06:21.975536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1173087061,"num_objects":10708,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10708,"num_whiteouts":0,"num_read":2482,"num_read_kb":1090930,"num_write":33245,"num_write_kb":1302990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":164450,"num_omap_keys":278,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33583,"ondisk_log_size":33583,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2375056,"kb_used_data":1792528,"kb_used_omap":12,"kb_used_meta":582451,"kb_avail":752599664,"statfs":{"total":773094113280,"available":770662055936,"internally_reserved":0,"allocated":1835548672,"data_stored":2346171322,"data_compressed":220026250,"data_compressed_allocated":252469248,"data_compressed_original":820019200,"omap_allocated":12750,"internal_metadata":596430386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36966366,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":37163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":164450,"num_omap_keys":278,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001994"},"pg_stats":[{"pgid":"3.3f","version":"38'348","reported_seq":2450,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844750+0000","last_change":"2024-03-19T03:00:04.520460+0000","last_active":"2024-03-19T03:06:12.844750+0000","last_peered":"2024-03-19T03:06:12.844750+0000","last_clean":"2024-03-19T03:06:12.844750+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T03:06:12.844750+0000","last_undegraded":"2024-03-19T03:06:12.844750+0000","last_fullsized":"2024-03-19T03:06:12.844750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:33.132110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045874986,"stat_sum":{"num_bytes":26339349,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":26,"num_read_kb":8207,"num_write":340,"num_write_kb":25903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"38'320","reported_seq":2424,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843903+0000","last_change":"2024-03-19T03:00:04.730981+0000","last_active":"2024-03-19T03:06:12.843903+0000","last_peered":"2024-03-19T03:06:12.843903+0000","last_clean":"2024-03-19T03:06:12.843903+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T03:06:12.843903+0000","last_undegraded":"2024-03-19T03:06:12.843903+0000","last_fullsized":"2024-03-19T03:06:12.843903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047342589999999997,"stat_sum":{"num_bytes":1430220,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":315,"num_write_kb":1566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"38'329","reported_seq":2439,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.847603+0000","last_change":"2024-03-19T03:00:04.294476+0000","last_active":"2024-03-19T03:06:12.847603+0000","last_peered":"2024-03-19T03:06:12.847603+0000","last_clean":"2024-03-19T03:06:12.847603+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T03:06:12.847603+0000","last_undegraded":"2024-03-19T03:06:12.847603+0000","last_fullsized":"2024-03-19T03:06:12.847603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069861674999999998,"stat_sum":{"num_bytes":15481473,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":33,"num_read_kb":8211,"num_write":321,"num_write_kb":23482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"38'334","reported_seq":2430,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843403+0000","last_change":"2024-03-19T03:00:04.461050+0000","last_active":"2024-03-19T03:06:12.843403+0000","last_peered":"2024-03-19T03:06:12.843403+0000","last_clean":"2024-03-19T03:06:12.843403+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T03:06:12.843403+0000","last_undegraded":"2024-03-19T03:06:12.843403+0000","last_fullsized":"2024-03-19T03:06:12.843403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054309962000000003,"stat_sum":{"num_bytes":20068260,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":19,"num_read_kb":8201,"num_write":329,"num_write_kb":19659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"38'352","reported_seq":2452,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844081+0000","last_change":"2024-03-19T03:00:04.518539+0000","last_active":"2024-03-19T03:06:12.844081+0000","last_peered":"2024-03-19T03:06:12.844081+0000","last_clean":"2024-03-19T03:06:12.844081+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T03:06:12.844081+0000","last_undegraded":"2024-03-19T03:06:12.844081+0000","last_fullsized":"2024-03-19T03:06:12.844081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:46.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051753202999999998,"stat_sum":{"num_bytes":6591355,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":25,"num_read_kb":4114,"num_write":347,"num_write_kb":6624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"38'337","reported_seq":2441,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.850148+0000","last_change":"2024-03-19T03:00:04.458228+0000","last_active":"2024-03-19T03:06:12.850148+0000","last_peered":"2024-03-19T03:06:12.850148+0000","last_clean":"2024-03-19T03:06:12.850148+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T03:06:12.850148+0000","last_undegraded":"2024-03-19T03:06:12.850148+0000","last_fullsized":"2024-03-19T03:06:12.850148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051735362,"stat_sum":{"num_bytes":12769638,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":28,"num_read_kb":4115,"num_write":331,"num_write_kb":12643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"38'349","reported_seq":2446,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849042+0000","last_change":"2024-03-19T03:00:04.518609+0000","last_active":"2024-03-19T03:06:12.849042+0000","last_peered":"2024-03-19T03:06:12.849042+0000","last_clean":"2024-03-19T03:06:12.849042+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T03:06:12.849042+0000","last_undegraded":"2024-03-19T03:06:12.849042+0000","last_fullsized":"2024-03-19T03:06:12.849042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:41.608998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051913059999999997,"stat_sum":{"num_bytes":23638346,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":344,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"38'307","reported_seq":2398,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.841913+0000","last_change":"2024-03-19T03:00:04.268010+0000","last_active":"2024-03-19T03:06:12.841913+0000","last_peered":"2024-03-19T03:06:12.841913+0000","last_clean":"2024-03-19T03:06:12.841913+0000","last_became_active":"2024-03-19T02:55:45.392555+0000","last_became_peered":"2024-03-19T02:55:45.392555+0000","last_unstale":"2024-03-19T03:06:12.841913+0000","last_undegraded":"2024-03-19T03:06:12.841913+0000","last_fullsized":"2024-03-19T03:06:12.841913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052882827,"stat_sum":{"num_bytes":15124789,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":21,"num_read_kb":4106,"num_write":297,"num_write_kb":14929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"38'333","reported_seq":2438,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.847807+0000","last_change":"2024-03-19T03:00:04.520474+0000","last_active":"2024-03-19T03:06:12.847807+0000","last_peered":"2024-03-19T03:06:12.847807+0000","last_clean":"2024-03-19T03:06:12.847807+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T03:06:12.847807+0000","last_undegraded":"2024-03-19T03:06:12.847807+0000","last_fullsized":"2024-03-19T03:06:12.847807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:42:29.088176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045138699999999997,"stat_sum":{"num_bytes":8352128,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":30,"num_read_kb":4119,"num_write":322,"num_write_kb":12425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"38'320","reported_seq":2421,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.841078+0000","last_change":"2024-03-19T03:00:04.469206+0000","last_active":"2024-03-19T03:06:12.841078+0000","last_peered":"2024-03-19T03:06:12.841078+0000","last_clean":"2024-03-19T03:06:12.841078+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T03:06:12.841078+0000","last_undegraded":"2024-03-19T03:06:12.841078+0000","last_fullsized":"2024-03-19T03:06:12.841078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.863780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033183580999999997,"stat_sum":{"num_bytes":17872088,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":8208,"num_write":302,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"38'311","reported_seq":2412,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.832859+0000","last_change":"2024-03-19T03:00:04.468803+0000","last_active":"2024-03-19T03:06:12.832859+0000","last_peered":"2024-03-19T03:06:12.832859+0000","last_clean":"2024-03-19T03:06:12.832859+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T03:06:12.832859+0000","last_undegraded":"2024-03-19T03:06:12.832859+0000","last_fullsized":"2024-03-19T03:06:12.832859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:05.106811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036038919000000003,"stat_sum":{"num_bytes":18621884,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":26,"num_read_kb":8210,"num_write":305,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"38'298","reported_seq":2392,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.838178+0000","last_change":"2024-03-19T03:00:04.233381+0000","last_active":"2024-03-19T03:06:12.838178+0000","last_peered":"2024-03-19T03:06:12.838178+0000","last_clean":"2024-03-19T03:06:12.838178+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T03:06:12.838178+0000","last_undegraded":"2024-03-19T03:06:12.838178+0000","last_fullsized":"2024-03-19T03:06:12.838178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016224016000000001,"stat_sum":{"num_bytes":10799086,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":18,"num_read_kb":10,"num_write":291,"num_write_kb":14795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"38'296","reported_seq":2398,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.837533+0000","last_change":"2024-03-19T03:00:04.638962+0000","last_active":"2024-03-19T03:06:12.837533+0000","last_peered":"2024-03-19T03:06:12.837533+0000","last_clean":"2024-03-19T03:06:12.837533+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T03:06:12.837533+0000","last_undegraded":"2024-03-19T03:06:12.837533+0000","last_fullsized":"2024-03-19T03:06:12.837533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061626093999999999,"stat_sum":{"num_bytes":18869857,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":26,"num_read_kb":4115,"num_write":285,"num_write_kb":26775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"38'337","reported_seq":2442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.848057+0000","last_change":"2024-03-19T03:00:04.294731+0000","last_active":"2024-03-19T03:06:12.848057+0000","last_peered":"2024-03-19T03:06:12.848057+0000","last_clean":"2024-03-19T03:06:12.848057+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:12.848057+0000","last_undegraded":"2024-03-19T03:06:12.848057+0000","last_fullsized":"2024-03-19T03:06:12.848057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.06936341,"stat_sum":{"num_bytes":7021822,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":330,"num_write_kb":6980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"38'358","reported_seq":2461,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.840788+0000","last_change":"2024-03-19T03:00:04.731871+0000","last_active":"2024-03-19T03:06:12.840788+0000","last_peered":"2024-03-19T03:06:12.840788+0000","last_clean":"2024-03-19T03:06:12.840788+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T03:06:12.840788+0000","last_undegraded":"2024-03-19T03:06:12.840788+0000","last_fullsized":"2024-03-19T03:06:12.840788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047989829999999997,"stat_sum":{"num_bytes":22976920,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":351,"num_write_kb":22623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"38'357","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849018+0000","last_change":"2024-03-19T03:00:04.463722+0000","last_active":"2024-03-19T03:06:12.849018+0000","last_peered":"2024-03-19T03:06:12.849018+0000","last_clean":"2024-03-19T03:06:12.849018+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T03:06:12.849018+0000","last_undegraded":"2024-03-19T03:06:12.849018+0000","last_fullsized":"2024-03-19T03:06:12.849018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054164370000000003,"stat_sum":{"num_bytes":10727184,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":27,"num_read_kb":4115,"num_write":346,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"38'352","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.846262+0000","last_change":"2024-03-19T03:00:04.463595+0000","last_active":"2024-03-19T03:06:12.846262+0000","last_peered":"2024-03-19T03:06:12.846262+0000","last_clean":"2024-03-19T03:06:12.846262+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T03:06:12.846262+0000","last_undegraded":"2024-03-19T03:06:12.846262+0000","last_fullsized":"2024-03-19T03:06:12.846262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054055032000000003,"stat_sum":{"num_bytes":10124226,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":32,"num_read_kb":4117,"num_write":343,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"38'300","reported_seq":2409,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844118+0000","last_change":"2024-03-19T03:00:04.265271+0000","last_active":"2024-03-19T03:06:12.844118+0000","last_peered":"2024-03-19T03:06:12.844118+0000","last_clean":"2024-03-19T03:06:12.844118+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T03:06:12.844118+0000","last_undegraded":"2024-03-19T03:06:12.844118+0000","last_fullsized":"2024-03-19T03:06:12.844118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.049230071,"stat_sum":{"num_bytes":7542726,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":289,"num_write_kb":7514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"38'329","reported_seq":2437,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849608+0000","last_change":"2024-03-19T03:00:04.518999+0000","last_active":"2024-03-19T03:06:12.849608+0000","last_peered":"2024-03-19T03:06:12.849608+0000","last_clean":"2024-03-19T03:06:12.849608+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T03:06:12.849608+0000","last_undegraded":"2024-03-19T03:06:12.849608+0000","last_fullsized":"2024-03-19T03:06:12.849608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:30.582720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050023194999999999,"stat_sum":{"num_bytes":14383098,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":32,"num_read_kb":16400,"num_write":321,"num_write_kb":18311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"38'360","reported_seq":2469,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843444+0000","last_change":"2024-03-19T03:00:04.469246+0000","last_active":"2024-03-19T03:06:12.843444+0000","last_peered":"2024-03-19T03:06:12.843444+0000","last_clean":"2024-03-19T03:06:12.843444+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T03:06:12.843444+0000","last_undegraded":"2024-03-19T03:06:12.843444+0000","last_fullsized":"2024-03-19T03:06:12.843444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:17:26.060817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033531875000000003,"stat_sum":{"num_bytes":23233989,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":33,"num_read_kb":12308,"num_write":352,"num_write_kb":31057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"38'329","reported_seq":2439,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:15.059416+0000","last_change":"2024-03-19T03:00:04.294471+0000","last_active":"2024-03-19T03:06:15.059416+0000","last_peered":"2024-03-19T03:06:15.059416+0000","last_clean":"2024-03-19T03:06:15.059416+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T03:06:15.059416+0000","last_undegraded":"2024-03-19T03:06:15.059416+0000","last_fullsized":"2024-03-19T03:06:15.059416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068442499000000004,"stat_sum":{"num_bytes":13700535,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":33,"num_read_kb":3602,"num_write":320,"num_write_kb":13502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"38'353","reported_seq":2457,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.850784+0000","last_change":"2024-03-19T03:00:04.460208+0000","last_active":"2024-03-19T03:06:12.850784+0000","last_peered":"2024-03-19T03:06:12.850784+0000","last_clean":"2024-03-19T03:06:12.850784+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T03:06:12.850784+0000","last_undegraded":"2024-03-19T03:06:12.850784+0000","last_fullsized":"2024-03-19T03:06:12.850784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053636608000000002,"stat_sum":{"num_bytes":9951503,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":340,"num_write_kb":13990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"38'332","reported_seq":2446,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:15.186896+0000","last_change":"2024-03-19T03:00:04.634734+0000","last_active":"2024-03-19T03:06:15.186896+0000","last_peered":"2024-03-19T03:06:15.186896+0000","last_clean":"2024-03-19T03:06:15.186896+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T03:06:15.186896+0000","last_undegraded":"2024-03-19T03:06:15.186896+0000","last_fullsized":"2024-03-19T03:06:15.186896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069022274999999994,"stat_sum":{"num_bytes":15469691,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":37,"num_read_kb":13386,"num_write":325,"num_write_kb":19374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"38'353","reported_seq":2468,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843543+0000","last_change":"2024-03-19T03:00:04.464949+0000","last_active":"2024-03-19T03:06:12.843543+0000","last_peered":"2024-03-19T03:06:12.843543+0000","last_clean":"2024-03-19T03:06:12.843543+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T03:06:12.843543+0000","last_undegraded":"2024-03-19T03:06:12.843543+0000","last_fullsized":"2024-03-19T03:06:12.843543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055361406000000002,"stat_sum":{"num_bytes":13836038,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":38,"num_read_kb":4325,"num_write":347,"num_write_kb":13690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"38'318","reported_seq":2423,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.837358+0000","last_change":"2024-03-19T03:00:04.636666+0000","last_active":"2024-03-19T03:06:12.837358+0000","last_peered":"2024-03-19T03:06:12.837358+0000","last_clean":"2024-03-19T03:06:12.837358+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T03:06:12.837358+0000","last_undegraded":"2024-03-19T03:06:12.837358+0000","last_fullsized":"2024-03-19T03:06:12.837358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066517528000000006,"stat_sum":{"num_bytes":14678298,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":314,"num_write_kb":14504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"38'372","reported_seq":2474,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.851502+0000","last_change":"2024-03-19T03:00:04.469451+0000","last_active":"2024-03-19T03:06:12.851502+0000","last_peered":"2024-03-19T03:06:12.851502+0000","last_clean":"2024-03-19T03:06:12.851502+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T03:06:12.851502+0000","last_undegraded":"2024-03-19T03:06:12.851502+0000","last_fullsized":"2024-03-19T03:06:12.851502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:08.415495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033380219000000003,"stat_sum":{"num_bytes":6267254,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":357,"num_write_kb":6310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"38'329","reported_seq":2432,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844355+0000","last_change":"2024-03-19T03:00:04.731007+0000","last_active":"2024-03-19T03:06:12.844355+0000","last_peered":"2024-03-19T03:06:12.844355+0000","last_clean":"2024-03-19T03:06:12.844355+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T03:06:12.844355+0000","last_undegraded":"2024-03-19T03:06:12.844355+0000","last_fullsized":"2024-03-19T03:06:12.844355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046987292,"stat_sum":{"num_bytes":20194837,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":27,"num_read_kb":4111,"num_write":325,"num_write_kb":23913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"38'316","reported_seq":2415,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.839282+0000","last_change":"2024-03-19T03:00:04.639714+0000","last_active":"2024-03-19T03:06:12.839282+0000","last_peered":"2024-03-19T03:06:12.839282+0000","last_clean":"2024-03-19T03:06:12.839282+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T03:06:12.839282+0000","last_undegraded":"2024-03-19T03:06:12.839282+0000","last_fullsized":"2024-03-19T03:06:12.839282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054969302999999997,"stat_sum":{"num_bytes":19046740,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":22,"num_read_kb":7383,"num_write":312,"num_write_kb":22038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"38'355","reported_seq":2468,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849532+0000","last_change":"2024-03-19T03:00:04.295066+0000","last_active":"2024-03-19T03:06:12.849532+0000","last_peered":"2024-03-19T03:06:12.849532+0000","last_clean":"2024-03-19T03:06:12.849532+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T03:06:12.849532+0000","last_undegraded":"2024-03-19T03:06:12.849532+0000","last_fullsized":"2024-03-19T03:06:12.849532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068635079000000002,"stat_sum":{"num_bytes":22225330,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":36,"num_read_kb":12306,"num_write":348,"num_write_kb":21889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"38'351","reported_seq":2464,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.846859+0000","last_change":"2024-03-19T03:00:04.460792+0000","last_active":"2024-03-19T03:06:12.846859+0000","last_peered":"2024-03-19T03:06:12.846859+0000","last_clean":"2024-03-19T03:06:12.846859+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T03:06:12.846859+0000","last_undegraded":"2024-03-19T03:06:12.846859+0000","last_fullsized":"2024-03-19T03:06:12.846859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054050365000000003,"stat_sum":{"num_bytes":13922214,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":36,"num_read_kb":4118,"num_write":345,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"38'406","reported_seq":2515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849533+0000","last_change":"2024-03-19T03:00:04.266403+0000","last_active":"2024-03-19T03:06:12.849533+0000","last_peered":"2024-03-19T03:06:12.849533+0000","last_clean":"2024-03-19T03:06:12.849533+0000","last_became_active":"2024-03-19T02:55:45.950634+0000","last_became_peered":"2024-03-19T02:55:45.950634+0000","last_unstale":"2024-03-19T03:06:12.849533+0000","last_undegraded":"2024-03-19T03:06:12.849533+0000","last_fullsized":"2024-03-19T03:06:12.849533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050892488,"stat_sum":{"num_bytes":11658804,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":27,"num_read_kb":4113,"num_write":395,"num_write_kb":15629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"38'369","reported_seq":2481,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844864+0000","last_change":"2024-03-19T03:00:04.467477+0000","last_active":"2024-03-19T03:06:12.844864+0000","last_peered":"2024-03-19T03:06:12.844864+0000","last_clean":"2024-03-19T03:06:12.844864+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T03:06:12.844864+0000","last_undegraded":"2024-03-19T03:06:12.844864+0000","last_fullsized":"2024-03-19T03:06:12.844864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:19.601901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035464137999999999,"stat_sum":{"num_bytes":11414841,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":36,"num_read_kb":4123,"num_write":359,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"38'329","reported_seq":2428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.840971+0000","last_change":"2024-03-19T03:00:04.233486+0000","last_active":"2024-03-19T03:06:12.840971+0000","last_peered":"2024-03-19T03:06:12.840971+0000","last_clean":"2024-03-19T03:06:12.840971+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T03:06:12.840971+0000","last_undegraded":"2024-03-19T03:06:12.840971+0000","last_fullsized":"2024-03-19T03:06:12.840971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016612378000000001,"stat_sum":{"num_bytes":15052974,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":23,"num_read_kb":4112,"num_write":322,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'49","reported_seq":113,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:14.773725+0000","last_change":"2024-03-19T02:54:55.205881+0000","last_active":"2024-03-19T03:05:14.773725+0000","last_peered":"2024-03-19T03:05:14.773725+0000","last_clean":"2024-03-19T03:05:14.773725+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T03:05:14.773725+0000","last_undegraded":"2024-03-19T03:05:14.773725+0000","last_fullsized":"2024-03-19T03:05:14.773725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":52,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"38'315","reported_seq":2417,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849561+0000","last_change":"2024-03-19T03:00:04.295011+0000","last_active":"2024-03-19T03:06:12.849561+0000","last_peered":"2024-03-19T03:06:12.849561+0000","last_clean":"2024-03-19T03:06:12.849561+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T03:06:12.849561+0000","last_undegraded":"2024-03-19T03:06:12.849561+0000","last_fullsized":"2024-03-19T03:06:12.849561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066278250999999996,"stat_sum":{"num_bytes":6674098,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":15,"num_write":306,"num_write_kb":6677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"38'198","reported_seq":268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:17.188393+0000","last_change":"2024-03-19T02:54:55.184298+0000","last_active":"2024-03-19T03:05:17.188393+0000","last_peered":"2024-03-19T03:05:17.188393+0000","last_clean":"2024-03-19T03:05:17.188393+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T03:05:17.188393+0000","last_undegraded":"2024-03-19T03:05:17.188393+0000","last_fullsized":"2024-03-19T03:05:17.188393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:48.832909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":61440,"num_write":198,"num_write_kb":16593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"38'31","reported_seq":98,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.744372+0000","last_change":"2024-03-19T03:06:19.744508+0000","last_active":"2024-03-19T03:06:19.744372+0000","last_peered":"2024-03-19T03:06:19.744372+0000","last_clean":"2024-03-19T03:06:19.744372+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T03:06:19.744372+0000","last_undegraded":"2024-03-19T03:06:19.744372+0000","last_fullsized":"2024-03-19T03:06:19.744372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'31","last_scrub_stamp":"2024-03-19T03:06:19.744110+0000","last_deep_scrub":"38'31","last_deep_scrub_stamp":"2024-03-19T03:06:19.744110+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.744110+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:05.113905+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":29,"num_write_kb":4131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27583,"num_omap_keys":47,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"38'394","reported_seq":2504,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.845962+0000","last_change":"2024-03-19T03:00:04.638741+0000","last_active":"2024-03-19T03:06:12.845962+0000","last_peered":"2024-03-19T03:06:12.845962+0000","last_clean":"2024-03-19T03:06:12.845962+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T03:06:12.845962+0000","last_undegraded":"2024-03-19T03:06:12.845962+0000","last_fullsized":"2024-03-19T03:06:12.845962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061005943,"stat_sum":{"num_bytes":22416031,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":34,"num_read_kb":16407,"num_write":386,"num_write_kb":31278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"38'751","reported_seq":815,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:10.651566+0000","last_change":"2024-03-19T02:54:55.175601+0000","last_active":"2024-03-19T03:06:10.651566+0000","last_peered":"2024-03-19T03:06:10.651566+0000","last_clean":"2024-03-19T03:06:10.651566+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T03:06:10.651566+0000","last_undegraded":"2024-03-19T03:06:10.651566+0000","last_fullsized":"2024-03-19T03:06:10.651566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":748,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"38'331","reported_seq":2437,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.845066+0000","last_change":"2024-03-19T03:00:04.731670+0000","last_active":"2024-03-19T03:06:12.845066+0000","last_peered":"2024-03-19T03:06:12.845066+0000","last_clean":"2024-03-19T03:06:12.845066+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T03:06:12.845066+0000","last_undegraded":"2024-03-19T03:06:12.845066+0000","last_fullsized":"2024-03-19T03:06:12.845066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048289845999999997,"stat_sum":{"num_bytes":20433584,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":31,"num_read_kb":16402,"num_write":319,"num_write_kb":24181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'8","reported_seq":62,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.733040+0000","last_change":"2024-03-19T02:54:55.182618+0000","last_active":"2024-03-19T03:05:09.733040+0000","last_peered":"2024-03-19T03:05:09.733040+0000","last_clean":"2024-03-19T03:05:09.733040+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T03:05:09.733040+0000","last_undegraded":"2024-03-19T03:05:09.733040+0000","last_fullsized":"2024-03-19T03:05:09.733040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:18:13.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"38'358","reported_seq":2460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849849+0000","last_change":"2024-03-19T03:00:04.266848+0000","last_active":"2024-03-19T03:06:12.849849+0000","last_peered":"2024-03-19T03:06:12.849849+0000","last_clean":"2024-03-19T03:06:12.849849+0000","last_became_active":"2024-03-19T02:55:45.403591+0000","last_became_peered":"2024-03-19T02:55:45.403591+0000","last_unstale":"2024-03-19T03:06:12.849849+0000","last_undegraded":"2024-03-19T03:06:12.849849+0000","last_fullsized":"2024-03-19T03:06:12.849849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050955334999999997,"stat_sum":{"num_bytes":22779940,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":32,"num_read_kb":10328,"num_write":348,"num_write_kb":22426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"38'1621","reported_seq":1682,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.053140+0000","last_change":"2024-03-19T02:54:55.183458+0000","last_active":"2024-03-19T03:06:01.053140+0000","last_peered":"2024-03-19T03:06:01.053140+0000","last_clean":"2024-03-19T03:06:01.053140+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T03:06:01.053140+0000","last_undegraded":"2024-03-19T03:06:01.053140+0000","last_fullsized":"2024-03-19T03:06:01.053140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":19565,"num_write":1617,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"38'316","reported_seq":2425,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849076+0000","last_change":"2024-03-19T03:00:04.295350+0000","last_active":"2024-03-19T03:06:12.849076+0000","last_peered":"2024-03-19T03:06:12.849076+0000","last_clean":"2024-03-19T03:06:12.849076+0000","last_became_active":"2024-03-19T02:55:45.952615+0000","last_became_peered":"2024-03-19T02:55:45.952615+0000","last_unstale":"2024-03-19T03:06:12.849076+0000","last_undegraded":"2024-03-19T03:06:12.849076+0000","last_fullsized":"2024-03-19T03:06:12.849076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067870387000000004,"stat_sum":{"num_bytes":9678845,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":28,"num_read_kb":4114,"num_write":307,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"38'335","reported_seq":413,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:18.329314+0000","last_change":"2024-03-19T03:06:18.329442+0000","last_active":"2024-03-19T03:06:18.329314+0000","last_peered":"2024-03-19T03:06:18.329314+0000","last_clean":"2024-03-19T03:06:18.329314+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T03:06:18.329314+0000","last_undegraded":"2024-03-19T03:06:18.329314+0000","last_fullsized":"2024-03-19T03:06:18.329314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'335","last_scrub_stamp":"2024-03-19T03:06:18.329106+0000","last_deep_scrub":"38'335","last_deep_scrub_stamp":"2024-03-19T03:06:18.329106+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.329106+0000","objects_scrubbed":5,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:05.059377+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196071,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":16407,"num_write":328,"num_write_kb":10141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"38'321","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849824+0000","last_change":"2024-03-19T03:00:04.233584+0000","last_active":"2024-03-19T03:06:12.849824+0000","last_peered":"2024-03-19T03:06:12.849824+0000","last_clean":"2024-03-19T03:06:12.849824+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T03:06:12.849824+0000","last_undegraded":"2024-03-19T03:06:12.849824+0000","last_fullsized":"2024-03-19T03:06:12.849824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016376714000000001,"stat_sum":{"num_bytes":15448536,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":23,"num_read_kb":4110,"num_write":308,"num_write_kb":19342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"38'161","reported_seq":245,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:54.120316+0000","last_change":"2024-03-19T02:54:55.191962+0000","last_active":"2024-03-19T03:05:54.120316+0000","last_peered":"2024-03-19T03:05:54.120316+0000","last_clean":"2024-03-19T03:05:54.120316+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T03:05:54.120316+0000","last_undegraded":"2024-03-19T03:05:54.120316+0000","last_fullsized":"2024-03-19T03:05:54.120316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:33.201733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":57372,"num_write":154,"num_write_kb":12387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"38'327","reported_seq":2429,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.844354+0000","last_change":"2024-03-19T03:00:04.517597+0000","last_active":"2024-03-19T03:06:12.844354+0000","last_peered":"2024-03-19T03:06:12.844354+0000","last_clean":"2024-03-19T03:06:12.844354+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T03:06:12.844354+0000","last_undegraded":"2024-03-19T03:06:12.844354+0000","last_fullsized":"2024-03-19T03:06:12.844354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:01:07.022928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053512995000000001,"stat_sum":{"num_bytes":2174562,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":27,"num_read_kb":19,"num_write":323,"num_write_kb":2244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"38'159","reported_seq":242,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:17.216384+0000","last_change":"2024-03-19T03:06:17.216450+0000","last_active":"2024-03-19T03:06:17.216384+0000","last_peered":"2024-03-19T03:06:17.216384+0000","last_clean":"2024-03-19T03:06:17.216384+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T03:06:17.216384+0000","last_undegraded":"2024-03-19T03:06:17.216384+0000","last_fullsized":"2024-03-19T03:06:17.216384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'159","last_scrub_stamp":"2024-03-19T03:06:17.216259+0000","last_deep_scrub":"38'159","last_deep_scrub_stamp":"2024-03-19T03:06:17.216259+0000","last_clean_scrub_stamp":"2024-03-19T03:06:17.216259+0000","objects_scrubbed":5,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:49.902547+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":41000,"num_write":152,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14584,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"38'340","reported_seq":2453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.850486+0000","last_change":"2024-03-19T03:00:04.293854+0000","last_active":"2024-03-19T03:06:12.850486+0000","last_peered":"2024-03-19T03:06:12.850486+0000","last_clean":"2024-03-19T03:06:12.850486+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T03:06:12.850486+0000","last_undegraded":"2024-03-19T03:06:12.850486+0000","last_fullsized":"2024-03-19T03:06:12.850486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068156319000000007,"stat_sum":{"num_bytes":18375725,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":37,"num_read_kb":16403,"num_write":329,"num_write_kb":26314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"37'1","reported_seq":56,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.450190+0000","last_change":"2024-03-19T02:54:55.199746+0000","last_active":"2024-03-19T03:05:10.450190+0000","last_peered":"2024-03-19T03:05:10.450190+0000","last_clean":"2024-03-19T03:05:10.450190+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T03:05:10.450190+0000","last_undegraded":"2024-03-19T03:05:10.450190+0000","last_fullsized":"2024-03-19T03:05:10.450190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"38'353","reported_seq":2458,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.845147+0000","last_change":"2024-03-19T03:00:04.518776+0000","last_active":"2024-03-19T03:06:12.845147+0000","last_peered":"2024-03-19T03:06:12.845147+0000","last_clean":"2024-03-19T03:06:12.845147+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T03:06:12.845147+0000","last_undegraded":"2024-03-19T03:06:12.845147+0000","last_fullsized":"2024-03-19T03:06:12.845147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.456673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051813761,"stat_sum":{"num_bytes":7359660,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":339,"num_write_kb":7364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"38'1","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:09.732969+0000","last_change":"2024-03-19T02:54:55.183279+0000","last_active":"2024-03-19T03:05:09.732969+0000","last_peered":"2024-03-19T03:05:09.732969+0000","last_clean":"2024-03-19T03:05:09.732969+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T03:05:09.732969+0000","last_undegraded":"2024-03-19T03:05:09.732969+0000","last_fullsized":"2024-03-19T03:05:09.732969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:07.650396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"38'346","reported_seq":2457,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.830762+0000","last_change":"2024-03-19T03:00:04.468344+0000","last_active":"2024-03-19T03:06:12.830762+0000","last_peered":"2024-03-19T03:06:12.830762+0000","last_clean":"2024-03-19T03:06:12.830762+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T03:06:12.830762+0000","last_undegraded":"2024-03-19T03:06:12.830762+0000","last_fullsized":"2024-03-19T03:06:12.830762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:34:09.179260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035899844,"stat_sum":{"num_bytes":22036853,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":37,"num_read_kb":12302,"num_write":335,"num_write_kb":21697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"38'128","reported_seq":216,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.060064+0000","last_change":"2024-03-19T02:54:55.199320+0000","last_active":"2024-03-19T03:06:01.060064+0000","last_peered":"2024-03-19T03:06:01.060064+0000","last_clean":"2024-03-19T03:06:01.060064+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T03:06:01.060064+0000","last_undegraded":"2024-03-19T03:06:01.060064+0000","last_fullsized":"2024-03-19T03:06:01.060064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":31075,"num_write":133,"num_write_kb":8289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"38'363","reported_seq":2471,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.845249+0000","last_change":"2024-03-19T03:00:04.518831+0000","last_active":"2024-03-19T03:06:12.845249+0000","last_peered":"2024-03-19T03:06:12.845249+0000","last_clean":"2024-03-19T03:06:12.845249+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T03:06:12.845249+0000","last_undegraded":"2024-03-19T03:06:12.845249+0000","last_fullsized":"2024-03-19T03:06:12.845249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:39.124926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051911553999999999,"stat_sum":{"num_bytes":11598304,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":33,"num_read_kb":12305,"num_write":357,"num_write_kb":19699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"38'37","reported_seq":110,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.601771+0000","last_change":"2024-03-19T03:06:20.601820+0000","last_active":"2024-03-19T03:06:20.601771+0000","last_peered":"2024-03-19T03:06:20.601771+0000","last_clean":"2024-03-19T03:06:20.601771+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T03:06:20.601771+0000","last_undegraded":"2024-03-19T03:06:20.601771+0000","last_fullsized":"2024-03-19T03:06:20.601771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'37","last_scrub_stamp":"2024-03-19T03:06:20.601707+0000","last_deep_scrub":"38'37","last_deep_scrub_stamp":"2024-03-19T03:06:20.601707+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.601707+0000","objects_scrubbed":9,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:51.991051+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":35,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75111,"num_omap_keys":125,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"38'301","reported_seq":2395,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.840819+0000","last_change":"2024-03-19T03:00:04.519069+0000","last_active":"2024-03-19T03:06:12.840819+0000","last_peered":"2024-03-19T03:06:12.840819+0000","last_clean":"2024-03-19T03:06:12.840819+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T03:06:12.840819+0000","last_undegraded":"2024-03-19T03:06:12.840819+0000","last_fullsized":"2024-03-19T03:06:12.840819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":301,"log_dups_size":0,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:27:49.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052296778000000002,"stat_sum":{"num_bytes":10391613,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":18,"num_read_kb":6155,"num_write":296,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"38'119","reported_seq":193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.575116+0000","last_change":"2024-03-19T03:06:19.575159+0000","last_active":"2024-03-19T03:06:19.575116+0000","last_peered":"2024-03-19T03:06:19.575116+0000","last_clean":"2024-03-19T03:06:19.575116+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T03:06:19.575116+0000","last_undegraded":"2024-03-19T03:06:19.575116+0000","last_fullsized":"2024-03-19T03:06:19.575116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'119","last_scrub_stamp":"2024-03-19T03:06:19.575016+0000","last_deep_scrub":"38'119","last_deep_scrub_stamp":"2024-03-19T03:06:19.575016+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.575016+0000","objects_scrubbed":3,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:16.168974+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":116,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9655,"num_omap_keys":17,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"38'332","reported_seq":2436,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.841930+0000","last_change":"2024-03-19T03:00:04.520259+0000","last_active":"2024-03-19T03:06:12.841930+0000","last_peered":"2024-03-19T03:06:12.841930+0000","last_clean":"2024-03-19T03:06:12.841930+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T03:06:12.841930+0000","last_undegraded":"2024-03-19T03:06:12.841930+0000","last_fullsized":"2024-03-19T03:06:12.841930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:46.091437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044895387000000002,"stat_sum":{"num_bytes":10001648,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":28,"num_read_kb":8211,"num_write":321,"num_write_kb":14027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"38'20","reported_seq":84,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:10.646550+0000","last_change":"2024-03-19T02:54:55.198356+0000","last_active":"2024-03-19T03:06:10.646550+0000","last_peered":"2024-03-19T03:06:10.646550+0000","last_clean":"2024-03-19T03:06:10.646550+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T03:06:10.646550+0000","last_undegraded":"2024-03-19T03:06:10.646550+0000","last_fullsized":"2024-03-19T03:06:10.646550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":20,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"38'368","reported_seq":2467,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.841789+0000","last_change":"2024-03-19T03:00:04.462041+0000","last_active":"2024-03-19T03:06:12.841789+0000","last_peered":"2024-03-19T03:06:12.841789+0000","last_clean":"2024-03-19T03:06:12.841789+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T03:06:12.841789+0000","last_undegraded":"2024-03-19T03:06:12.841789+0000","last_fullsized":"2024-03-19T03:06:12.841789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055101615,"stat_sum":{"num_bytes":11152492,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":22,"num_read_kb":4110,"num_write":358,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"38'108","reported_seq":173,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:18.615640+0000","last_change":"2024-03-19T03:06:18.615729+0000","last_active":"2024-03-19T03:06:18.615640+0000","last_peered":"2024-03-19T03:06:18.615640+0000","last_clean":"2024-03-19T03:06:18.615640+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T03:06:18.615640+0000","last_undegraded":"2024-03-19T03:06:18.615640+0000","last_fullsized":"2024-03-19T03:06:18.615640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'108","last_scrub_stamp":"2024-03-19T03:06:18.615428+0000","last_deep_scrub":"38'108","last_deep_scrub_stamp":"2024-03-19T03:06:18.615428+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.615428+0000","objects_scrubbed":3,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:33.816723+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":108,"num_write_kb":4216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10273,"num_omap_keys":17,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'336","reported_seq":2442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.847634+0000","last_change":"2024-03-19T03:00:04.517906+0000","last_active":"2024-03-19T03:06:12.847634+0000","last_peered":"2024-03-19T03:06:12.847634+0000","last_clean":"2024-03-19T03:06:12.847634+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T03:06:12.847634+0000","last_undegraded":"2024-03-19T03:06:12.847634+0000","last_fullsized":"2024-03-19T03:06:12.847634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:39.429284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.051414362999999998,"stat_sum":{"num_bytes":15817335,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":30,"num_read_kb":4113,"num_write":329,"num_write_kb":19678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":211,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:10.447511+0000","last_change":"2024-03-19T02:54:52.044343+0000","last_active":"2024-03-19T03:05:10.447511+0000","last_peered":"2024-03-19T03:05:10.447511+0000","last_clean":"2024-03-19T03:05:10.447511+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T03:05:10.447511+0000","last_undegraded":"2024-03-19T03:05:10.447511+0000","last_fullsized":"2024-03-19T03:05:10.447511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:50.857665+0000","last_clean_scrub_stamp":"2024-03-19T02:54:50.857665+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:48.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"38'1698","reported_seq":1757,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.060186+0000","last_change":"2024-03-19T02:54:55.183002+0000","last_active":"2024-03-19T03:06:01.060186+0000","last_peered":"2024-03-19T03:06:01.060186+0000","last_clean":"2024-03-19T03:06:01.060186+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T03:06:01.060186+0000","last_undegraded":"2024-03-19T03:06:01.060186+0000","last_fullsized":"2024-03-19T03:06:01.060186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5356,"num_write":1694,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"38'332","reported_seq":2434,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843529+0000","last_change":"2024-03-19T03:00:04.637745+0000","last_active":"2024-03-19T03:06:12.843529+0000","last_peered":"2024-03-19T03:06:12.843529+0000","last_clean":"2024-03-19T03:06:12.843529+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T03:06:12.843529+0000","last_undegraded":"2024-03-19T03:06:12.843529+0000","last_fullsized":"2024-03-19T03:06:12.843529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063353052000000007,"stat_sum":{"num_bytes":11364310,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4110,"num_write":322,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'187","reported_seq":253,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.165979+0000","last_change":"2024-03-19T02:54:55.198409+0000","last_active":"2024-03-19T03:05:15.165979+0000","last_peered":"2024-03-19T03:05:15.165979+0000","last_clean":"2024-03-19T03:05:15.165979+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T03:05:15.165979+0000","last_undegraded":"2024-03-19T03:05:15.165979+0000","last_fullsized":"2024-03-19T03:05:15.165979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:25:00.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":181,"num_write_kb":12402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"38'446","reported_seq":2557,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.384015+0000","last_change":"2024-03-19T03:06:19.384052+0000","last_active":"2024-03-19T03:06:19.384015+0000","last_peered":"2024-03-19T03:06:19.384015+0000","last_clean":"2024-03-19T03:06:19.384015+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T03:06:19.384015+0000","last_undegraded":"2024-03-19T03:06:19.384015+0000","last_fullsized":"2024-03-19T03:06:19.384015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'446","last_scrub_stamp":"2024-03-19T03:06:19.383952+0000","last_deep_scrub":"38'446","last_deep_scrub_stamp":"2024-03-19T03:06:19.383952+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.383952+0000","objects_scrubbed":163,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:49.812214+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0.068696631999999994,"stat_sum":{"num_bytes":14176427,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":313,"num_write_kb":14010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"38'94","reported_seq":243,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:51.859738+0000","last_change":"2024-03-19T02:54:55.194814+0000","last_active":"2024-03-19T03:05:51.859738+0000","last_peered":"2024-03-19T03:05:51.859738+0000","last_clean":"2024-03-19T03:05:51.859738+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T03:05:51.859738+0000","last_undegraded":"2024-03-19T03:05:51.859738+0000","last_fullsized":"2024-03-19T03:05:51.859738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":581,"num_write":90,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"38'332","reported_seq":2435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.846556+0000","last_change":"2024-03-19T03:00:04.732387+0000","last_active":"2024-03-19T03:06:12.846556+0000","last_peered":"2024-03-19T03:06:12.846556+0000","last_clean":"2024-03-19T03:06:12.846556+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T03:06:12.846556+0000","last_undegraded":"2024-03-19T03:06:12.846556+0000","last_fullsized":"2024-03-19T03:06:12.846556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044152405999999998,"stat_sum":{"num_bytes":9810356,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":324,"num_write_kb":9748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'61","reported_seq":126,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.922339+0000","last_change":"2024-03-19T02:54:55.200302+0000","last_active":"2024-03-19T03:05:20.922339+0000","last_peered":"2024-03-19T03:05:20.922339+0000","last_clean":"2024-03-19T03:05:20.922339+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T03:05:20.922339+0000","last_undegraded":"2024-03-19T03:05:20.922339+0000","last_fullsized":"2024-03-19T03:05:20.922339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":60,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"38'365","reported_seq":2470,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.842557+0000","last_change":"2024-03-19T03:00:04.233217+0000","last_active":"2024-03-19T03:06:12.842557+0000","last_peered":"2024-03-19T03:06:12.842557+0000","last_clean":"2024-03-19T03:06:12.842557+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T03:06:12.842557+0000","last_undegraded":"2024-03-19T03:06:12.842557+0000","last_fullsized":"2024-03-19T03:06:12.842557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016334565999999998,"stat_sum":{"num_bytes":7144435,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":28,"num_read_kb":4112,"num_write":358,"num_write_kb":7164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"37'363","reported_seq":419,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:20.047081+0000","last_change":"2024-03-19T02:54:55.204859+0000","last_active":"2024-03-19T03:05:20.047081+0000","last_peered":"2024-03-19T03:05:20.047081+0000","last_clean":"2024-03-19T03:05:20.047081+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T03:05:20.047081+0000","last_undegraded":"2024-03-19T03:05:20.047081+0000","last_fullsized":"2024-03-19T03:05:20.047081+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":360,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'341","reported_seq":2446,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.843713+0000","last_change":"2024-03-19T03:00:04.461266+0000","last_active":"2024-03-19T03:06:12.843713+0000","last_peered":"2024-03-19T03:06:12.843713+0000","last_clean":"2024-03-19T03:06:12.843713+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T03:06:12.843713+0000","last_undegraded":"2024-03-19T03:06:12.843713+0000","last_fullsized":"2024-03-19T03:06:12.843713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054510613999999999,"stat_sum":{"num_bytes":18185428,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":28,"num_read_kb":9793,"num_write":337,"num_write_kb":22033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'364","reported_seq":424,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:16.763732+0000","last_change":"2024-03-19T02:54:55.204317+0000","last_active":"2024-03-19T03:05:16.763732+0000","last_peered":"2024-03-19T03:05:16.763732+0000","last_clean":"2024-03-19T03:05:16.763732+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T03:05:16.763732+0000","last_undegraded":"2024-03-19T03:05:16.763732+0000","last_fullsized":"2024-03-19T03:05:16.763732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:44.272026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":360,"num_write_kb":8331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"38'356","reported_seq":2459,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.847028+0000","last_change":"2024-03-19T03:00:04.730764+0000","last_active":"2024-03-19T03:06:12.847028+0000","last_peered":"2024-03-19T03:06:12.847028+0000","last_clean":"2024-03-19T03:06:12.847028+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T03:06:12.847028+0000","last_undegraded":"2024-03-19T03:06:12.847028+0000","last_fullsized":"2024-03-19T03:06:12.847028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047353098000000003,"stat_sum":{"num_bytes":7011110,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":27,"num_read_kb":4110,"num_write":348,"num_write_kb":6996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'272","reported_seq":349,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.342586+0000","last_change":"2024-03-19T03:06:19.342638+0000","last_active":"2024-03-19T03:06:19.342586+0000","last_peered":"2024-03-19T03:06:19.342586+0000","last_clean":"2024-03-19T03:06:19.342586+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T03:06:19.342586+0000","last_undegraded":"2024-03-19T03:06:19.342586+0000","last_fullsized":"2024-03-19T03:06:19.342586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'272","last_scrub_stamp":"2024-03-19T03:06:19.342467+0000","last_deep_scrub":"38'272","last_deep_scrub_stamp":"2024-03-19T03:06:19.342467+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.342467+0000","objects_scrubbed":2,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:20:18.043251+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":638,"num_write":618,"num_write_kb":2118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6878,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"38'312","reported_seq":2406,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.849962+0000","last_change":"2024-03-19T03:00:04.266141+0000","last_active":"2024-03-19T03:06:12.849962+0000","last_peered":"2024-03-19T03:06:12.849962+0000","last_clean":"2024-03-19T03:06:12.849962+0000","last_became_active":"2024-03-19T02:55:45.402992+0000","last_became_peered":"2024-03-19T02:55:45.402992+0000","last_unstale":"2024-03-19T03:06:12.849962+0000","last_undegraded":"2024-03-19T03:06:12.849962+0000","last_fullsized":"2024-03-19T03:06:12.849962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.050031514999999999,"stat_sum":{"num_bytes":18622985,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":24,"num_read_kb":16,"num_write":307,"num_write_kb":22442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'147","reported_seq":212,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.498499+0000","last_change":"2024-03-19T02:54:55.181149+0000","last_active":"2024-03-19T03:05:40.498499+0000","last_peered":"2024-03-19T03:05:40.498499+0000","last_clean":"2024-03-19T03:05:40.498499+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T03:05:40.498499+0000","last_undegraded":"2024-03-19T03:05:40.498499+0000","last_fullsized":"2024-03-19T03:05:40.498499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:33.788188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":9370,"num_write":147,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"38'360","reported_seq":2476,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.848600+0000","last_change":"2024-03-19T03:00:04.295347+0000","last_active":"2024-03-19T03:06:12.848600+0000","last_peered":"2024-03-19T03:06:12.848600+0000","last_clean":"2024-03-19T03:06:12.848600+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:12.848600+0000","last_undegraded":"2024-03-19T03:06:12.848600+0000","last_fullsized":"2024-03-19T03:06:12.848600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068525034999999998,"stat_sum":{"num_bytes":35227376,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":40,"num_read_kb":16714,"num_write":347,"num_write_kb":34590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'17","reported_seq":76,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.056765+0000","last_change":"2024-03-19T02:54:55.199074+0000","last_active":"2024-03-19T03:06:01.056765+0000","last_peered":"2024-03-19T03:06:01.056765+0000","last_clean":"2024-03-19T03:06:01.056765+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T03:06:01.056765+0000","last_undegraded":"2024-03-19T03:06:01.056765+0000","last_fullsized":"2024-03-19T03:06:01.056765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:46.108926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":20,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"38'346","reported_seq":2453,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.814032+0000","last_change":"2024-03-19T03:00:04.233647+0000","last_active":"2024-03-19T03:06:12.814032+0000","last_peered":"2024-03-19T03:06:12.814032+0000","last_clean":"2024-03-19T03:06:12.814032+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T03:06:12.814032+0000","last_undegraded":"2024-03-19T03:06:12.814032+0000","last_fullsized":"2024-03-19T03:06:12.814032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016349961999999999,"stat_sum":{"num_bytes":23588150,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":30,"num_read_kb":20,"num_write":344,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"38'264","reported_seq":322,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:01.056846+0000","last_change":"2024-03-19T02:54:55.201734+0000","last_active":"2024-03-19T03:06:01.056846+0000","last_peered":"2024-03-19T03:06:01.056846+0000","last_clean":"2024-03-19T03:06:01.056846+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T03:06:01.056846+0000","last_undegraded":"2024-03-19T03:06:01.056846+0000","last_fullsized":"2024-03-19T03:06:01.056846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":264,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"38'354","reported_seq":2465,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.845782+0000","last_change":"2024-03-19T03:00:04.639224+0000","last_active":"2024-03-19T03:06:12.845782+0000","last_peered":"2024-03-19T03:06:12.845782+0000","last_clean":"2024-03-19T03:06:12.845782+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T03:06:12.845782+0000","last_undegraded":"2024-03-19T03:06:12.845782+0000","last_fullsized":"2024-03-19T03:06:12.845782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061753209000000003,"stat_sum":{"num_bytes":19421231,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":34,"num_read_kb":12304,"num_write":344,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"38'77","reported_seq":137,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:11.876794+0000","last_change":"2024-03-19T02:54:55.206773+0000","last_active":"2024-03-19T03:06:11.876794+0000","last_peered":"2024-03-19T03:06:11.876794+0000","last_clean":"2024-03-19T03:06:11.876794+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T03:06:11.876794+0000","last_undegraded":"2024-03-19T03:06:11.876794+0000","last_fullsized":"2024-03-19T03:06:11.876794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":76,"num_write_kb":12383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"38'339","reported_seq":2439,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:15.186942+0000","last_change":"2024-03-19T03:00:04.639280+0000","last_active":"2024-03-19T03:06:15.186942+0000","last_peered":"2024-03-19T03:06:15.186942+0000","last_clean":"2024-03-19T03:06:15.186942+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T03:06:15.186942+0000","last_undegraded":"2024-03-19T03:06:15.186942+0000","last_fullsized":"2024-03-19T03:06:15.186942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061740428,"stat_sum":{"num_bytes":14078096,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":23,"num_read_kb":8205,"num_write":330,"num_write_kb":13929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"38'282","reported_seq":352,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:10.932741+0000","last_change":"2024-03-19T02:54:55.181208+0000","last_active":"2024-03-19T03:06:10.932741+0000","last_peered":"2024-03-19T03:06:10.932741+0000","last_clean":"2024-03-19T03:06:10.932741+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T03:06:10.932741+0000","last_undegraded":"2024-03-19T03:06:10.932741+0000","last_fullsized":"2024-03-19T03:06:10.932741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:22.827919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":40970,"num_write":274,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"38'347","reported_seq":2449,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.835170+0000","last_change":"2024-03-19T03:00:04.467727+0000","last_active":"2024-03-19T03:06:12.835170+0000","last_peered":"2024-03-19T03:06:12.835170+0000","last_clean":"2024-03-19T03:06:12.835170+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T03:06:12.835170+0000","last_undegraded":"2024-03-19T03:06:12.835170+0000","last_fullsized":"2024-03-19T03:06:12.835170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:03.908237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035412037,"stat_sum":{"num_bytes":11182535,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":26,"num_read_kb":4109,"num_write":343,"num_write_kb":11053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"38'317","reported_seq":2420,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.839784+0000","last_change":"2024-03-19T03:00:04.460570+0000","last_active":"2024-03-19T03:06:12.839784+0000","last_peered":"2024-03-19T03:06:12.839784+0000","last_clean":"2024-03-19T03:06:12.839784+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T03:06:12.839784+0000","last_undegraded":"2024-03-19T03:06:12.839784+0000","last_fullsized":"2024-03-19T03:06:12.839784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053825845999999997,"stat_sum":{"num_bytes":7076024,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":303,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"37'88","reported_seq":152,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:15.432014+0000","last_change":"2024-03-19T02:54:55.181514+0000","last_active":"2024-03-19T03:05:15.432014+0000","last_peered":"2024-03-19T03:05:15.432014+0000","last_clean":"2024-03-19T03:05:15.432014+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T03:05:15.432014+0000","last_undegraded":"2024-03-19T03:05:15.432014+0000","last_fullsized":"2024-03-19T03:05:15.432014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:41.253813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":92,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"38'351","reported_seq":2451,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.838539+0000","last_change":"2024-03-19T03:00:04.460533+0000","last_active":"2024-03-19T03:06:12.838539+0000","last_peered":"2024-03-19T03:06:12.838539+0000","last_clean":"2024-03-19T03:06:12.838539+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T03:06:12.838539+0000","last_undegraded":"2024-03-19T03:06:12.838539+0000","last_fullsized":"2024-03-19T03:06:12.838539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.053805551,"stat_sum":{"num_bytes":27334536,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":24,"num_read_kb":7741,"num_write":341,"num_write_kb":30967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"38'2315","reported_seq":2387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:05:40.196171+0000","last_change":"2024-03-19T02:54:55.182578+0000","last_active":"2024-03-19T03:05:40.196171+0000","last_peered":"2024-03-19T03:05:40.196171+0000","last_clean":"2024-03-19T03:05:40.196171+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T03:05:40.196171+0000","last_undegraded":"2024-03-19T03:05:40.196171+0000","last_fullsized":"2024-03-19T03:05:40.196171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:58:31.905640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":49170,"num_write":2306,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"38'330","reported_seq":2438,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.850997+0000","last_change":"2024-03-19T03:00:04.294754+0000","last_active":"2024-03-19T03:06:12.850997+0000","last_peered":"2024-03-19T03:06:12.850997+0000","last_clean":"2024-03-19T03:06:12.850997+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T03:06:12.850997+0000","last_undegraded":"2024-03-19T03:06:12.850997+0000","last_fullsized":"2024-03-19T03:06:12.850997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067674389000000001,"stat_sum":{"num_bytes":14508603,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":31,"num_read_kb":8211,"num_write":323,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"38'1551","reported_seq":1619,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:02.738021+0000","last_change":"2024-03-19T02:54:55.201806+0000","last_active":"2024-03-19T03:06:02.738021+0000","last_peered":"2024-03-19T03:06:02.738021+0000","last_clean":"2024-03-19T03:06:02.738021+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T03:06:02.738021+0000","last_undegraded":"2024-03-19T03:06:02.738021+0000","last_fullsized":"2024-03-19T03:06:02.738021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:54.149570+0000","last_clean_scrub_stamp":"2024-03-19T02:54:54.149570+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":42561,"num_write":1549,"num_write_kb":13574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"38'337","reported_seq":2435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:12.839979+0000","last_change":"2024-03-19T03:00:04.636091+0000","last_active":"2024-03-19T03:06:12.839979+0000","last_peered":"2024-03-19T03:06:12.839979+0000","last_clean":"2024-03-19T03:06:12.839979+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T03:06:12.839979+0000","last_undegraded":"2024-03-19T03:06:12.839979+0000","last_fullsized":"2024-03-19T03:06:12.839979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:54:55.159487+0000","last_clean_scrub_stamp":"2024-03-19T02:54:55.159487+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065375344000000002,"stat_sum":{"num_bytes":15008616,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":21,"num_read_kb":8204,"num_write":333,"num_write_kb":14782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"38'154","reported_seq":240,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:18.199549+0000","last_change":"2024-03-19T03:06:18.199587+0000","last_active":"2024-03-19T03:06:18.199549+0000","last_peered":"2024-03-19T03:06:18.199549+0000","last_clean":"2024-03-19T03:06:18.199549+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T03:06:18.199549+0000","last_undegraded":"2024-03-19T03:06:18.199549+0000","last_fullsized":"2024-03-19T03:06:18.199549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'154","last_scrub_stamp":"2024-03-19T03:06:18.199476+0000","last_deep_scrub":"38'154","last_deep_scrub_stamp":"2024-03-19T03:06:18.199476+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.199476+0000","objects_scrubbed":6,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.265804+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":45076,"num_write":147,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20366,"num_omap_keys":34,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":939466941,"num_objects":10564,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10564,"num_whiteouts":0,"num_read":1809,"num_read_kb":363000,"num_write":21148,"num_write_kb":1062314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1709039616,"data_stored":1877393786,"data_compressed":164809216,"data_compressed_allocated":182280192,"data_compressed_original":405618688,"omap_allocated":0,"internal_metadata":0},"log_size":21805,"ondisk_log_size":21805,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":233029752,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":557,"num_read_kb":727830,"num_write":11938,"num_write_kb":238260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":164450,"num_omap_keys":278,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122970112,"data_stored":466059504,"data_compressed":55152242,"data_compressed_allocated":70098944,"data_compressed_original":413253632,"omap_allocated":0,"internal_metadata":0},"log_size":11661,"ondisk_log_size":11661,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574988,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332180,"kb_used_data":242700,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94039660,"statfs":{"total":96636764160,"available":96296611840,"internally_reserved":0,"allocated":248524800,"data_stored":332229150,"data_compressed":32896566,"data_compressed_allocated":38154240,"data_compressed_original":130297856,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574989,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283920,"kb_used_data":194504,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94087920,"statfs":{"total":96636764160,"available":96346030080,"internally_reserved":0,"allocated":199172096,"data_stored":260537624,"data_compressed":19587928,"data_compressed_allocated":23076864,"data_compressed_original":89665536,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574989,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365964,"kb_used_data":254276,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94005876,"statfs":{"total":96636764160,"available":96262017024,"internally_reserved":0,"allocated":260378624,"data_stored":321477105,"data_compressed":24311902,"data_compressed_allocated":28250112,"data_compressed_original":97251328,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325212,"kb_used_data":253780,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94046628,"statfs":{"total":96636764160,"available":96303747072,"internally_reserved":0,"allocated":259870720,"data_stored":336209440,"data_compressed":37546180,"data_compressed_allocated":42463232,"data_compressed_original":127148032,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574990,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339460,"kb_used_data":276348,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94032380,"statfs":{"total":96636764160,"available":96289157120,"internally_reserved":0,"allocated":282980352,"data_stored":343276606,"data_compressed":27693759,"data_compressed_allocated":31715328,"data_compressed_original":100061184,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":184580,"kb_used_data":139516,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94187260,"statfs":{"total":96636764160,"available":96447754240,"internally_reserved":0,"allocated":142864384,"data_stored":174161585,"data_compressed":16362216,"data_compressed_allocated":18694144,"data_compressed_original":54902784,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212888,"kb_used_data":167824,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94158952,"statfs":{"total":96636764160,"available":96418766848,"internally_reserved":0,"allocated":171851776,"data_stored":244570190,"data_compressed":28880142,"data_compressed_allocated":33181696,"data_compressed_original":111534080,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574989,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330852,"kb_used_data":263580,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94040988,"statfs":{"total":96636764160,"available":96297971712,"internally_reserved":0,"allocated":269905920,"data_stored":333709622,"data_compressed":32747557,"data_compressed_allocated":36933632,"data_compressed_original":109158400,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12947456,"data_stored":50332286,"data_compressed":6253471,"data_compressed_allocated":7770112,"data_compressed_original":45166592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14815232,"data_stored":64722093,"data_compressed":8311031,"data_compressed_allocated":10489856,"data_compressed_original":60407808,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":21505709,"data_compressed":2290904,"data_compressed_allocated":3022848,"data_compressed_original":17223680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11972608,"data_stored":50866509,"data_compressed":6550764,"data_compressed_allocated":8245248,"data_compressed_original":47149056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14008320,"data_stored":62914686,"data_compressed":7417000,"data_compressed_allocated":9461760,"data_compressed_original":58380288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21200896,"data_stored":64722023,"data_compressed":7283115,"data_compressed_allocated":9195520,"data_compressed_original":52723712,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20590592,"data_stored":71303840,"data_compressed":7958306,"data_compressed_allocated":10170368,"data_compressed_original":60891136,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":79692358,"data_compressed":9087651,"data_compressed_allocated":11743232,"data_compressed_original":71311360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256466944,"data_stored":282594806,"data_compressed":26461690,"data_compressed_allocated":29118464,"data_compressed_original":63418368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156610560,"data_stored":179655935,"data_compressed":20569111,"data_compressed_allocated":22691840,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135131136,"data_stored":152463714,"data_compressed":14071312,"data_compressed_allocated":15671296,"data_compressed_original":37679104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270581760,"data_stored":292217935,"data_compressed":21142995,"data_compressed_allocated":23470080,"data_compressed_original":52912128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245436416,"data_stored":273102592,"data_compressed":30129180,"data_compressed_allocated":33001472,"data_compressed_original":68767744,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238751744,"data_stored":256562920,"data_compressed":17028787,"data_compressed_allocated":19054592,"data_compressed_original":44527616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178155520,"data_stored":189041622,"data_compressed":11629622,"data_compressed_allocated":12906496,"data_compressed_original":28774400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":227905536,"data_stored":251754262,"data_compressed":23776519,"data_compressed_allocated":26365952,"data_compressed_original":58413056,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:06:22.440 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.440 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.440 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.441 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.442 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.443 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.444 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.445 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.446 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.447 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.448 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.449 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.450 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-19T02:54:50.857665+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=50, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.450 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.450 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.450 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.450 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.451 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.452 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.453 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-19T02:54:54.149570+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=54, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.454 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-19T02:54:55.159487+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=54, tm_sec=55, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=3, tm_min=6, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T03:06:22.455 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-19T03:06:32.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:32.329 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:42.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:06:42.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.667+0000 7f36b1000640 1 Processor -- start 2024-03-19T03:06:42.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.667+0000 7f36b1000640 1 -- start start 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.668+0000 7f36b1000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 0x7f36ac06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36b1000640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f36ac1593d0 0x7f36ac0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36b1000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36ac0c9a80 con 0x7f36ac0c6470 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36b1000640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f36ac0c6c90 con 0x7f36ac1593d0 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36b1000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f36ac159910 con 0x7f36ac152bf0 2024-03-19T03:06:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 0x7f36ac06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36aa800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f36ac1593d0 0x7f36ac0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9400640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f36ac0c6470 0x7f36ac0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:33436/0 (socket says 172.21.15.92:33436) 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9400640 1 -- 172.21.15.92:0/4216346240 learned_addr learned my addr 172.21.15.92:0/4216346240 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9e00640 1 -- 172.21.15.92:0/4216346240 >> v2:172.21.15.159:3300/0 conn(0x7f36ac1593d0 msgr2=0x7f36ac0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9e00640 1 --2- 172.21.15.92:0/4216346240 >> v2:172.21.15.159:3300/0 conn(0x7f36ac1593d0 0x7f36ac0c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9e00640 1 -- 172.21.15.92:0/4216346240 >> v1:172.21.15.159:6789/0 conn(0x7f36ac0c6470 legacy=0x7f36ac0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a9e00640 1 -- 172.21.15.92:0/4216346240 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac0c8650 con 0x7f36ac152bf0 2024-03-19T03:06:42.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.669+0000 7f36a8a00640 1 -- 172.21.15.92:0/4216346240 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028845824 0 0) 0x7f36ac0c9a80 con 0x7f36ac0c6470 2024-03-19T03:06:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36a9e00640 1 --2- 172.21.15.92:0/4216346240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 0x7f36ac06eef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36940097a0 tx=0x7f369402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2f4d56ba3ff0c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36a8a00640 1 -- 172.21.15.92:0/4216346240 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940315a0 con 0x7f36ac152bf0 2024-03-19T03:06:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36a8a00640 1 -- 172.21.15.92:0/4216346240 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3694037070 con 0x7f36ac152bf0 2024-03-19T03:06:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36a8a00640 1 -- 172.21.15.92:0/4216346240 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694031960 con 0x7f36ac152bf0 2024-03-19T03:06:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36b1000640 1 -- 172.21.15.92:0/4216346240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 msgr2=0x7f36ac06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.671+0000 7f36b1000640 1 --2- 172.21.15.92:0/4216346240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 0x7f36ac06eef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36940097a0 tx=0x7f369402b840 comp rx=0 tx=0).stop 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 -- 172.21.15.92:0/4216346240 shutdown_connections 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 --2- 172.21.15.92:0/4216346240 >> v2:172.21.15.159:3300/0 conn(0x7f36ac1593d0 0x7f36ac0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 --2- 172.21.15.92:0/4216346240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac152bf0 0x7f36ac06eef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 -- 172.21.15.92:0/4216346240 >> 172.21.15.92:0/4216346240 conn(0x7f36ac0c8d90 msgr2=0x7f36ac0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 -- 172.21.15.92:0/4216346240 shutdown_connections 2024-03-19T03:06:42.675 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.672+0000 7f36b1000640 1 -- 172.21.15.92:0/4216346240 wait complete. 2024-03-19T03:06:42.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.673+0000 7f36b1000640 1 Processor -- start 2024-03-19T03:06:42.677 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 -- start start 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36ac152bf0 0x7f36ac1f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 0x7f36ac072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac06ea60 con 0x7f36ac152bf0 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac06e4a0 con 0x7f36ac0c6470 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.674+0000 7f36b1000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f36ac06e6f0 con 0x7f36ac1593d0 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36a9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.678 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36a9e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40518/0 (socket says 172.21.15.92:40518) 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36a9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36ac152bf0 0x7f36ac1f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36a9e00640 1 -- 172.21.15.92:0/1872132529 learned_addr learned my addr 172.21.15.92:0/1872132529 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 0x7f36ac072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 -- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 msgr2=0x7f36ac144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 -- 172.21.15.92:0/1872132529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36ac152bf0 msgr2=0x7f36ac1f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36ac152bf0 0x7f36ac1f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.679 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 -- 172.21.15.92:0/1872132529 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac1451e0 con 0x7f36ac1593d0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36a9e00640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.675+0000 7f36aa800640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 0x7f36ac072580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36a400e820 tx=0x7f36a400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.676+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4017070 con 0x7f36ac1593d0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.676+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a400c730 con 0x7f36ac1593d0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.676+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a400cb00 con 0x7f36ac1593d0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.676+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ac1454c0 con 0x7f36ac1593d0 2024-03-19T03:06:42.680 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.676+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ac1f4520 con 0x7f36ac1593d0 2024-03-19T03:06:42.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.677+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36ac0684c0 con 0x7f36ac1593d0 2024-03-19T03:06:42.681 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.678+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f36a401a070 con 0x7f36ac1593d0 2024-03-19T03:06:42.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.679+0000 7f36a2000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 0x7f368806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.679+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f36a408aa00 con 0x7f36ac1593d0 2024-03-19T03:06:42.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.679+0000 7f36a9e00640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 0x7f368806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.682 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.679+0000 7f36a9e00640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 0x7f368806e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3694000c00 tx=0x7f369403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.684 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.683+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f36a4058cd0 con 0x7f36ac1593d0 2024-03-19T03:06:42.877 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.875+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 --> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36ac14cc70 con 0x7f368806bc20 2024-03-19T03:06:42.885 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.884+0000 7f36a2000640 1 -- 172.21.15.92:0/1872132529 <== mgr.4108 v2:172.21.15.159:6832/697081936 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257697 (secure 0 0 0) 0x7f36ac14cc70 con 0x7f368806bc20 2024-03-19T03:06:42.889 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:42.891 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-19T03:06:42.892 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 msgr2=0x7f368806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 0x7f368806e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3694000c00 tx=0x7f369403d040 comp rx=0 tx=0).stop 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 msgr2=0x7f36ac072580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 0x7f36ac072580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36a400e820 tx=0x7f36a400ece0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 shutdown_connections 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f368806bc20 0x7f368806e0e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f36ac1593d0 0x7f36ac072580 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36ac152bf0 0x7f36ac1f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 --2- 172.21.15.92:0/1872132529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f36ac0c6470 0x7f36ac144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 >> 172.21.15.92:0/1872132529 conn(0x7f36ac0c8d90 msgr2=0x7f36ac14f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:42.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 shutdown_connections 2024-03-19T03:06:42.894 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:42.889+0000 7f36b1000640 1 -- 172.21.15.92:0/1872132529 wait complete. 2024-03-19T03:06:42.909 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-19T03:06:41.979002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1173474310,"num_objects":10708,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10708,"num_whiteouts":0,"num_read":2482,"num_read_kb":1090930,"num_write":33265,"num_write_kb":1303379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":582147,"num_omap_keys":982,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42257,"ondisk_log_size":42257,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2396080,"kb_used_data":1792752,"kb_used_omap":12,"kb_used_meta":603251,"kb_avail":752578640,"statfs":{"total":773094113280,"available":770640527360,"internally_reserved":0,"allocated":1835778048,"data_stored":2346945820,"data_compressed":220099090,"data_compressed_allocated":252567552,"data_compressed_original":820658176,"omap_allocated":12750,"internal_metadata":617729586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":305342,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":127947,"num_omap_keys":216,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"3.3f","version":"38'498","reported_seq":2616,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:34.955341+0000","last_change":"2024-03-19T03:06:34.955370+0000","last_active":"2024-03-19T03:06:34.955341+0000","last_peered":"2024-03-19T03:06:34.955341+0000","last_clean":"2024-03-19T03:06:34.955341+0000","last_became_active":"2024-03-19T02:54:56.197655+0000","last_became_peered":"2024-03-19T02:54:56.197655+0000","last_unstale":"2024-03-19T03:06:34.955341+0000","last_undegraded":"2024-03-19T03:06:34.955341+0000","last_fullsized":"2024-03-19T03:06:34.955341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'498","last_scrub_stamp":"2024-03-19T03:06:34.955279+0000","last_deep_scrub":"38'498","last_deep_scrub_stamp":"2024-03-19T03:06:34.955279+0000","last_clean_scrub_stamp":"2024-03-19T03:06:34.955279+0000","objects_scrubbed":182,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:26.018737+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0.045874986,"stat_sum":{"num_bytes":26339349,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":26,"num_read_kb":8207,"num_write":340,"num_write_kb":25903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"38'463","reported_seq":2581,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:29.511100+0000","last_change":"2024-03-19T03:06:29.511156+0000","last_active":"2024-03-19T03:06:29.511100+0000","last_peered":"2024-03-19T03:06:29.511100+0000","last_clean":"2024-03-19T03:06:29.511100+0000","last_became_active":"2024-03-19T02:54:56.213502+0000","last_became_peered":"2024-03-19T02:54:56.213502+0000","last_unstale":"2024-03-19T03:06:29.511100+0000","last_undegraded":"2024-03-19T03:06:29.511100+0000","last_fullsized":"2024-03-19T03:06:29.511100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'463","last_scrub_stamp":"2024-03-19T03:06:29.511016+0000","last_deep_scrub":"38'463","last_deep_scrub_stamp":"2024-03-19T03:06:29.511016+0000","last_clean_scrub_stamp":"2024-03-19T03:06:29.511016+0000","objects_scrubbed":173,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:11.269602+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0.047342589999999997,"stat_sum":{"num_bytes":1430220,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":315,"num_write_kb":1566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"38'458","reported_seq":2582,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.465568+0000","last_change":"2024-03-19T03:06:25.465601+0000","last_active":"2024-03-19T03:06:25.465568+0000","last_peered":"2024-03-19T03:06:25.465568+0000","last_clean":"2024-03-19T03:06:25.465568+0000","last_became_active":"2024-03-19T02:54:56.212541+0000","last_became_peered":"2024-03-19T02:54:56.212541+0000","last_unstale":"2024-03-19T03:06:25.465568+0000","last_undegraded":"2024-03-19T03:06:25.465568+0000","last_fullsized":"2024-03-19T03:06:25.465568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'458","last_scrub_stamp":"2024-03-19T03:06:25.465502+0000","last_deep_scrub":"38'458","last_deep_scrub_stamp":"2024-03-19T03:06:25.465502+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.465502+0000","objects_scrubbed":157,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:12.085149+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0.069861674999999998,"stat_sum":{"num_bytes":15481473,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":33,"num_read_kb":8211,"num_write":321,"num_write_kb":23482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"38'484","reported_seq":2595,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:31.908418+0000","last_change":"2024-03-19T03:06:31.908469+0000","last_active":"2024-03-19T03:06:31.908418+0000","last_peered":"2024-03-19T03:06:31.908418+0000","last_clean":"2024-03-19T03:06:31.908418+0000","last_became_active":"2024-03-19T02:54:56.195127+0000","last_became_peered":"2024-03-19T02:54:56.195127+0000","last_unstale":"2024-03-19T03:06:31.908418+0000","last_undegraded":"2024-03-19T03:06:31.908418+0000","last_fullsized":"2024-03-19T03:06:31.908418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'484","last_scrub_stamp":"2024-03-19T03:06:31.908349+0000","last_deep_scrub":"38'484","last_deep_scrub_stamp":"2024-03-19T03:06:31.908349+0000","last_clean_scrub_stamp":"2024-03-19T03:06:31.908349+0000","objects_scrubbed":187,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:15:56.279545+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0.054309962000000003,"stat_sum":{"num_bytes":20068260,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":19,"num_read_kb":8201,"num_write":329,"num_write_kb":19659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"38'474","reported_seq":2589,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:31.982508+0000","last_change":"2024-03-19T03:06:31.982532+0000","last_active":"2024-03-19T03:06:31.982508+0000","last_peered":"2024-03-19T03:06:31.982508+0000","last_clean":"2024-03-19T03:06:31.982508+0000","last_became_active":"2024-03-19T02:54:56.193791+0000","last_became_peered":"2024-03-19T02:54:56.193791+0000","last_unstale":"2024-03-19T03:06:31.982508+0000","last_undegraded":"2024-03-19T03:06:31.982508+0000","last_fullsized":"2024-03-19T03:06:31.982508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'474","last_scrub_stamp":"2024-03-19T03:06:31.982452+0000","last_deep_scrub":"38'474","last_deep_scrub_stamp":"2024-03-19T03:06:31.982452+0000","last_clean_scrub_stamp":"2024-03-19T03:06:31.982452+0000","objects_scrubbed":151,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:18.741165+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0.051753202999999998,"stat_sum":{"num_bytes":6591355,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":25,"num_read_kb":4114,"num_write":347,"num_write_kb":6624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"38'466","reported_seq":2584,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:36.830310+0000","last_change":"2024-03-19T03:06:36.830359+0000","last_active":"2024-03-19T03:06:36.830310+0000","last_peered":"2024-03-19T03:06:36.830310+0000","last_clean":"2024-03-19T03:06:36.830310+0000","last_became_active":"2024-03-19T02:54:56.213375+0000","last_became_peered":"2024-03-19T02:54:56.213375+0000","last_unstale":"2024-03-19T03:06:36.830310+0000","last_undegraded":"2024-03-19T03:06:36.830310+0000","last_fullsized":"2024-03-19T03:06:36.830310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'466","last_scrub_stamp":"2024-03-19T03:06:36.830236+0000","last_deep_scrub":"38'466","last_deep_scrub_stamp":"2024-03-19T03:06:36.830236+0000","last_clean_scrub_stamp":"2024-03-19T03:06:36.830236+0000","objects_scrubbed":161,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:00.560498+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0.051735362,"stat_sum":{"num_bytes":12769638,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":28,"num_read_kb":4115,"num_write":331,"num_write_kb":12643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"38'469","reported_seq":2580,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:36.958640+0000","last_change":"2024-03-19T03:06:36.958679+0000","last_active":"2024-03-19T03:06:36.958640+0000","last_peered":"2024-03-19T03:06:36.958640+0000","last_clean":"2024-03-19T03:06:36.958640+0000","last_became_active":"2024-03-19T02:54:56.199120+0000","last_became_peered":"2024-03-19T02:54:56.199120+0000","last_unstale":"2024-03-19T03:06:36.958640+0000","last_undegraded":"2024-03-19T03:06:36.958640+0000","last_fullsized":"2024-03-19T03:06:36.958640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'469","last_scrub_stamp":"2024-03-19T03:06:36.958580+0000","last_deep_scrub":"38'469","last_deep_scrub_stamp":"2024-03-19T03:06:36.958580+0000","last_clean_scrub_stamp":"2024-03-19T03:06:36.958580+0000","objects_scrubbed":147,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:47.540861+0000","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0.051913059999999997,"stat_sum":{"num_bytes":23638346,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":344,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"38'441","reported_seq":2546,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.833114+0000","last_change":"2024-03-19T03:06:23.833214+0000","last_active":"2024-03-19T03:06:23.833114+0000","last_peered":"2024-03-19T03:06:23.833114+0000","last_clean":"2024-03-19T03:06:23.833114+0000","last_became_active":"2024-03-19T02:55:45.392555+0000","last_became_peered":"2024-03-19T02:55:45.392555+0000","last_unstale":"2024-03-19T03:06:23.833114+0000","last_undegraded":"2024-03-19T03:06:23.833114+0000","last_fullsized":"2024-03-19T03:06:23.833114+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'441","last_scrub_stamp":"2024-03-19T03:06:23.832957+0000","last_deep_scrub":"38'441","last_deep_scrub_stamp":"2024-03-19T03:06:23.832957+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.832957+0000","objects_scrubbed":169,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:56:30.513936+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0.052882827,"stat_sum":{"num_bytes":15124789,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":21,"num_read_kb":4106,"num_write":297,"num_write_kb":14929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"38'471","reported_seq":2591,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:35.873239+0000","last_change":"2024-03-19T03:06:35.873305+0000","last_active":"2024-03-19T03:06:35.873239+0000","last_peered":"2024-03-19T03:06:35.873239+0000","last_clean":"2024-03-19T03:06:35.873239+0000","last_became_active":"2024-03-19T02:54:56.198429+0000","last_became_peered":"2024-03-19T02:54:56.198429+0000","last_unstale":"2024-03-19T03:06:35.873239+0000","last_undegraded":"2024-03-19T03:06:35.873239+0000","last_fullsized":"2024-03-19T03:06:35.873239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'471","last_scrub_stamp":"2024-03-19T03:06:35.873169+0000","last_deep_scrub":"38'471","last_deep_scrub_stamp":"2024-03-19T03:06:35.873169+0000","last_clean_scrub_stamp":"2024-03-19T03:06:35.873169+0000","objects_scrubbed":170,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:51:50.893477+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0.045138699999999997,"stat_sum":{"num_bytes":8352128,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":30,"num_read_kb":4119,"num_write":322,"num_write_kb":12425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"38'438","reported_seq":2553,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:30.645140+0000","last_change":"2024-03-19T03:06:30.645185+0000","last_active":"2024-03-19T03:06:30.645140+0000","last_peered":"2024-03-19T03:06:30.645140+0000","last_clean":"2024-03-19T03:06:30.645140+0000","last_became_active":"2024-03-19T02:54:56.197399+0000","last_became_peered":"2024-03-19T02:54:56.197399+0000","last_unstale":"2024-03-19T03:06:30.645140+0000","last_undegraded":"2024-03-19T03:06:30.645140+0000","last_fullsized":"2024-03-19T03:06:30.645140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'438","last_scrub_stamp":"2024-03-19T03:06:30.645068+0000","last_deep_scrub":"38'438","last_deep_scrub_stamp":"2024-03-19T03:06:30.645068+0000","last_clean_scrub_stamp":"2024-03-19T03:06:30.645068+0000","objects_scrubbed":140,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:33.342841+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0.033183580999999997,"stat_sum":{"num_bytes":17872088,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":8208,"num_write":302,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"38'424","reported_seq":2539,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:29.639331+0000","last_change":"2024-03-19T03:06:29.639381+0000","last_active":"2024-03-19T03:06:29.639331+0000","last_peered":"2024-03-19T03:06:29.639331+0000","last_clean":"2024-03-19T03:06:29.639331+0000","last_became_active":"2024-03-19T02:54:56.196865+0000","last_became_peered":"2024-03-19T02:54:56.196865+0000","last_unstale":"2024-03-19T03:06:29.639331+0000","last_undegraded":"2024-03-19T03:06:29.639331+0000","last_fullsized":"2024-03-19T03:06:29.639331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'424","last_scrub_stamp":"2024-03-19T03:06:29.639232+0000","last_deep_scrub":"38'424","last_deep_scrub_stamp":"2024-03-19T03:06:29.639232+0000","last_clean_scrub_stamp":"2024-03-19T03:06:29.639232+0000","objects_scrubbed":136,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:52.524118+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0.036038919000000003,"stat_sum":{"num_bytes":18621884,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":26,"num_read_kb":8210,"num_write":305,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"38'415","reported_seq":2522,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.756428+0000","last_change":"2024-03-19T03:06:21.756493+0000","last_active":"2024-03-19T03:06:21.756428+0000","last_peered":"2024-03-19T03:06:21.756428+0000","last_clean":"2024-03-19T03:06:21.756428+0000","last_became_active":"2024-03-19T02:54:56.186930+0000","last_became_peered":"2024-03-19T02:54:56.186930+0000","last_unstale":"2024-03-19T03:06:21.756428+0000","last_undegraded":"2024-03-19T03:06:21.756428+0000","last_fullsized":"2024-03-19T03:06:21.756428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'415","last_scrub_stamp":"2024-03-19T03:06:21.756357+0000","last_deep_scrub":"38'415","last_deep_scrub_stamp":"2024-03-19T03:06:21.756357+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.756357+0000","objects_scrubbed":149,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:41:57.620217+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0.016224016000000001,"stat_sum":{"num_bytes":10799086,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":18,"num_read_kb":10,"num_write":291,"num_write_kb":14795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"38'413","reported_seq":2528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.963490+0000","last_change":"2024-03-19T03:06:23.963551+0000","last_active":"2024-03-19T03:06:23.963490+0000","last_peered":"2024-03-19T03:06:23.963490+0000","last_clean":"2024-03-19T03:06:23.963490+0000","last_became_active":"2024-03-19T02:54:56.191660+0000","last_became_peered":"2024-03-19T02:54:56.191660+0000","last_unstale":"2024-03-19T03:06:23.963490+0000","last_undegraded":"2024-03-19T03:06:23.963490+0000","last_fullsized":"2024-03-19T03:06:23.963490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'413","last_scrub_stamp":"2024-03-19T03:06:23.963380+0000","last_deep_scrub":"38'413","last_deep_scrub_stamp":"2024-03-19T03:06:23.963380+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.963380+0000","objects_scrubbed":143,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:42.552734+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0.061626093999999999,"stat_sum":{"num_bytes":18869857,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":26,"num_read_kb":4115,"num_write":285,"num_write_kb":26775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"38'479","reported_seq":2599,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.429409+0000","last_change":"2024-03-19T03:06:24.429409+0000","last_active":"2024-03-19T03:06:24.429409+0000","last_peered":"2024-03-19T03:06:24.429409+0000","last_clean":"2024-03-19T03:06:24.429409+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:24.429409+0000","last_undegraded":"2024-03-19T03:06:24.429409+0000","last_fullsized":"2024-03-19T03:06:24.429409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'479","last_scrub_stamp":"2024-03-19T03:06:24.429257+0000","last_deep_scrub":"38'479","last_deep_scrub_stamp":"2024-03-19T03:06:24.429257+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.429257+0000","objects_scrubbed":161,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:33.714266+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0.06936341,"stat_sum":{"num_bytes":7021822,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":330,"num_write_kb":6980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"38'519","reported_seq":2638,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.548679+0000","last_change":"2024-03-19T03:06:24.548679+0000","last_active":"2024-03-19T03:06:24.548679+0000","last_peered":"2024-03-19T03:06:24.548679+0000","last_clean":"2024-03-19T03:06:24.548679+0000","last_became_active":"2024-03-19T02:54:56.200828+0000","last_became_peered":"2024-03-19T02:54:56.200828+0000","last_unstale":"2024-03-19T03:06:24.548679+0000","last_undegraded":"2024-03-19T03:06:24.548679+0000","last_fullsized":"2024-03-19T03:06:24.548679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'519","last_scrub_stamp":"2024-03-19T03:06:24.548479+0000","last_deep_scrub":"38'519","last_deep_scrub_stamp":"2024-03-19T03:06:24.548479+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.548479+0000","objects_scrubbed":195,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:04.198232+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0.047989829999999997,"stat_sum":{"num_bytes":22976920,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":351,"num_write_kb":22623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"38'494","reported_seq":2612,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:33.862937+0000","last_change":"2024-03-19T03:06:33.862937+0000","last_active":"2024-03-19T03:06:33.862937+0000","last_peered":"2024-03-19T03:06:33.862937+0000","last_clean":"2024-03-19T03:06:33.862937+0000","last_became_active":"2024-03-19T02:54:56.200639+0000","last_became_peered":"2024-03-19T02:54:56.200639+0000","last_unstale":"2024-03-19T03:06:33.862937+0000","last_undegraded":"2024-03-19T03:06:33.862937+0000","last_fullsized":"2024-03-19T03:06:33.862937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'494","last_scrub_stamp":"2024-03-19T03:06:33.862784+0000","last_deep_scrub":"38'494","last_deep_scrub_stamp":"2024-03-19T03:06:33.862784+0000","last_clean_scrub_stamp":"2024-03-19T03:06:33.862784+0000","objects_scrubbed":166,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:15.104713+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0.054164370000000003,"stat_sum":{"num_bytes":10727184,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":27,"num_read_kb":4115,"num_write":346,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"38'498","reported_seq":2621,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:32.812350+0000","last_change":"2024-03-19T03:06:32.812350+0000","last_active":"2024-03-19T03:06:32.812350+0000","last_peered":"2024-03-19T03:06:32.812350+0000","last_clean":"2024-03-19T03:06:32.812350+0000","last_became_active":"2024-03-19T02:54:56.199596+0000","last_became_peered":"2024-03-19T02:54:56.199596+0000","last_unstale":"2024-03-19T03:06:32.812350+0000","last_undegraded":"2024-03-19T03:06:32.812350+0000","last_fullsized":"2024-03-19T03:06:32.812350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'498","last_scrub_stamp":"2024-03-19T03:06:32.812210+0000","last_deep_scrub":"38'498","last_deep_scrub_stamp":"2024-03-19T03:06:32.812210+0000","last_clean_scrub_stamp":"2024-03-19T03:06:32.812210+0000","objects_scrubbed":173,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:12.071603+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0.054055032000000003,"stat_sum":{"num_bytes":10124226,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":32,"num_read_kb":4117,"num_write":343,"num_write_kb":14164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"38'418","reported_seq":2541,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.733831+0000","last_change":"2024-03-19T03:06:22.733831+0000","last_active":"2024-03-19T03:06:22.733831+0000","last_peered":"2024-03-19T03:06:22.733831+0000","last_clean":"2024-03-19T03:06:22.733831+0000","last_became_active":"2024-03-19T02:54:56.188242+0000","last_became_peered":"2024-03-19T02:54:56.188242+0000","last_unstale":"2024-03-19T03:06:22.733831+0000","last_undegraded":"2024-03-19T03:06:22.733831+0000","last_fullsized":"2024-03-19T03:06:22.733831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'418","last_scrub_stamp":"2024-03-19T03:06:22.733567+0000","last_deep_scrub":"38'418","last_deep_scrub_stamp":"2024-03-19T03:06:22.733567+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.733567+0000","objects_scrubbed":137,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:21.178701+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0.049230071,"stat_sum":{"num_bytes":7542726,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":289,"num_write_kb":7514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"38'457","reported_seq":2580,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.061228+0000","last_change":"2024-03-19T03:06:23.061257+0000","last_active":"2024-03-19T03:06:23.061228+0000","last_peered":"2024-03-19T03:06:23.061228+0000","last_clean":"2024-03-19T03:06:23.061228+0000","last_became_active":"2024-03-19T02:54:56.194929+0000","last_became_peered":"2024-03-19T02:54:56.194929+0000","last_unstale":"2024-03-19T03:06:23.061228+0000","last_undegraded":"2024-03-19T03:06:23.061228+0000","last_fullsized":"2024-03-19T03:06:23.061228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'457","last_scrub_stamp":"2024-03-19T03:06:23.061169+0000","last_deep_scrub":"38'457","last_deep_scrub_stamp":"2024-03-19T03:06:23.061169+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.061169+0000","objects_scrubbed":163,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:50.206700+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0.050023194999999999,"stat_sum":{"num_bytes":14383098,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":32,"num_read_kb":16400,"num_write":321,"num_write_kb":18311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"38'495","reported_seq":2619,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.736896+0000","last_change":"2024-03-19T03:06:26.736968+0000","last_active":"2024-03-19T03:06:26.736896+0000","last_peered":"2024-03-19T03:06:26.736896+0000","last_clean":"2024-03-19T03:06:26.736896+0000","last_became_active":"2024-03-19T02:54:56.192576+0000","last_became_peered":"2024-03-19T02:54:56.192576+0000","last_unstale":"2024-03-19T03:06:26.736896+0000","last_undegraded":"2024-03-19T03:06:26.736896+0000","last_fullsized":"2024-03-19T03:06:26.736896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'495","last_scrub_stamp":"2024-03-19T03:06:26.736791+0000","last_deep_scrub":"38'495","last_deep_scrub_stamp":"2024-03-19T03:06:26.736791+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.736791+0000","objects_scrubbed":162,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:24.907743+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0.033531875000000003,"stat_sum":{"num_bytes":23233989,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":33,"num_read_kb":12308,"num_write":352,"num_write_kb":31057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"38'469","reported_seq":2593,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.514595+0000","last_change":"2024-03-19T03:06:26.514659+0000","last_active":"2024-03-19T03:06:26.514595+0000","last_peered":"2024-03-19T03:06:26.514595+0000","last_clean":"2024-03-19T03:06:26.514595+0000","last_became_active":"2024-03-19T02:54:56.212952+0000","last_became_peered":"2024-03-19T02:54:56.212952+0000","last_unstale":"2024-03-19T03:06:26.514595+0000","last_undegraded":"2024-03-19T03:06:26.514595+0000","last_fullsized":"2024-03-19T03:06:26.514595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'469","last_scrub_stamp":"2024-03-19T03:06:26.514494+0000","last_deep_scrub":"38'469","last_deep_scrub_stamp":"2024-03-19T03:06:26.514494+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.514494+0000","objects_scrubbed":164,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:09.960379+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0.068442499000000004,"stat_sum":{"num_bytes":13700535,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":33,"num_read_kb":3602,"num_write":320,"num_write_kb":13502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"38'492","reported_seq":2610,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.722117+0000","last_change":"2024-03-19T03:06:28.722165+0000","last_active":"2024-03-19T03:06:28.722117+0000","last_peered":"2024-03-19T03:06:28.722117+0000","last_clean":"2024-03-19T03:06:28.722117+0000","last_became_active":"2024-03-19T02:54:56.189999+0000","last_became_peered":"2024-03-19T02:54:56.189999+0000","last_unstale":"2024-03-19T03:06:28.722117+0000","last_undegraded":"2024-03-19T03:06:28.722117+0000","last_fullsized":"2024-03-19T03:06:28.722117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'492","last_scrub_stamp":"2024-03-19T03:06:28.722047+0000","last_deep_scrub":"38'492","last_deep_scrub_stamp":"2024-03-19T03:06:28.722047+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.722047+0000","objects_scrubbed":168,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:30:25.864198+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0.053636608000000002,"stat_sum":{"num_bytes":9951503,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":340,"num_write_kb":13990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"38'461","reported_seq":2589,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.029351+0000","last_change":"2024-03-19T03:06:23.029425+0000","last_active":"2024-03-19T03:06:23.029351+0000","last_peered":"2024-03-19T03:06:23.029351+0000","last_clean":"2024-03-19T03:06:23.029351+0000","last_became_active":"2024-03-19T02:54:56.190319+0000","last_became_peered":"2024-03-19T02:54:56.190319+0000","last_unstale":"2024-03-19T03:06:23.029351+0000","last_undegraded":"2024-03-19T03:06:23.029351+0000","last_fullsized":"2024-03-19T03:06:23.029351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'461","last_scrub_stamp":"2024-03-19T03:06:23.029186+0000","last_deep_scrub":"38'461","last_deep_scrub_stamp":"2024-03-19T03:06:23.029186+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.029186+0000","objects_scrubbed":158,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:14:44.257690+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0.069022274999999994,"stat_sum":{"num_bytes":15469691,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":37,"num_read_kb":13386,"num_write":325,"num_write_kb":19374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"38'489","reported_seq":2618,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:29.764414+0000","last_change":"2024-03-19T03:06:29.764481+0000","last_active":"2024-03-19T03:06:29.764414+0000","last_peered":"2024-03-19T03:06:29.764414+0000","last_clean":"2024-03-19T03:06:29.764414+0000","last_became_active":"2024-03-19T02:54:56.193019+0000","last_became_peered":"2024-03-19T02:54:56.193019+0000","last_unstale":"2024-03-19T03:06:29.764414+0000","last_undegraded":"2024-03-19T03:06:29.764414+0000","last_fullsized":"2024-03-19T03:06:29.764414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'489","last_scrub_stamp":"2024-03-19T03:06:29.764311+0000","last_deep_scrub":"38'489","last_deep_scrub_stamp":"2024-03-19T03:06:29.764311+0000","last_clean_scrub_stamp":"2024-03-19T03:06:29.764311+0000","objects_scrubbed":173,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:47:48.972155+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.055361406000000002,"stat_sum":{"num_bytes":13836038,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":38,"num_read_kb":4325,"num_write":347,"num_write_kb":13690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"38'454","reported_seq":2573,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.042312+0000","last_change":"2024-03-19T03:06:26.042394+0000","last_active":"2024-03-19T03:06:26.042312+0000","last_peered":"2024-03-19T03:06:26.042312+0000","last_clean":"2024-03-19T03:06:26.042312+0000","last_became_active":"2024-03-19T02:54:56.193722+0000","last_became_peered":"2024-03-19T02:54:56.193722+0000","last_unstale":"2024-03-19T03:06:26.042312+0000","last_undegraded":"2024-03-19T03:06:26.042312+0000","last_fullsized":"2024-03-19T03:06:26.042312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'454","last_scrub_stamp":"2024-03-19T03:06:26.042113+0000","last_deep_scrub":"38'454","last_deep_scrub_stamp":"2024-03-19T03:06:26.042113+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.042113+0000","objects_scrubbed":156,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:55:33.351395+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0.066517528000000006,"stat_sum":{"num_bytes":14678298,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":314,"num_write_kb":14504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"38'515","reported_seq":2632,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.641774+0000","last_change":"2024-03-19T03:06:28.641854+0000","last_active":"2024-03-19T03:06:28.641774+0000","last_peered":"2024-03-19T03:06:28.641774+0000","last_clean":"2024-03-19T03:06:28.641774+0000","last_became_active":"2024-03-19T02:54:56.195776+0000","last_became_peered":"2024-03-19T02:54:56.195776+0000","last_unstale":"2024-03-19T03:06:28.641774+0000","last_undegraded":"2024-03-19T03:06:28.641774+0000","last_fullsized":"2024-03-19T03:06:28.641774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'515","last_scrub_stamp":"2024-03-19T03:06:28.641678+0000","last_deep_scrub":"38'515","last_deep_scrub_stamp":"2024-03-19T03:06:28.641678+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.641678+0000","objects_scrubbed":173,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:44.972237+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0.033380219000000003,"stat_sum":{"num_bytes":6267254,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":357,"num_write_kb":6310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"38'485","reported_seq":2603,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.558474+0000","last_change":"2024-03-19T03:06:28.558508+0000","last_active":"2024-03-19T03:06:28.558474+0000","last_peered":"2024-03-19T03:06:28.558474+0000","last_clean":"2024-03-19T03:06:28.558474+0000","last_became_active":"2024-03-19T02:54:56.212068+0000","last_became_peered":"2024-03-19T02:54:56.212068+0000","last_unstale":"2024-03-19T03:06:28.558474+0000","last_undegraded":"2024-03-19T03:06:28.558474+0000","last_fullsized":"2024-03-19T03:06:28.558474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'485","last_scrub_stamp":"2024-03-19T03:06:28.558406+0000","last_deep_scrub":"38'485","last_deep_scrub_stamp":"2024-03-19T03:06:28.558406+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.558406+0000","objects_scrubbed":184,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:51.241235+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0.046987292,"stat_sum":{"num_bytes":20194837,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":27,"num_read_kb":4111,"num_write":325,"num_write_kb":23913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"38'444","reported_seq":2557,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.017132+0000","last_change":"2024-03-19T03:06:27.017242+0000","last_active":"2024-03-19T03:06:27.017132+0000","last_peered":"2024-03-19T03:06:27.017132+0000","last_clean":"2024-03-19T03:06:27.017132+0000","last_became_active":"2024-03-19T02:54:56.194898+0000","last_became_peered":"2024-03-19T02:54:56.194898+0000","last_unstale":"2024-03-19T03:06:27.017132+0000","last_undegraded":"2024-03-19T03:06:27.017132+0000","last_fullsized":"2024-03-19T03:06:27.017132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'444","last_scrub_stamp":"2024-03-19T03:06:27.016931+0000","last_deep_scrub":"38'444","last_deep_scrub_stamp":"2024-03-19T03:06:27.016931+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.016931+0000","objects_scrubbed":160,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:44:00.363942+0000","scrub_duration":282,"objects_trimmed":0,"snaptrim_duration":0.054969302999999997,"stat_sum":{"num_bytes":19046740,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":22,"num_read_kb":7383,"num_write":312,"num_write_kb":22038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"38'485","reported_seq":2612,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.529559+0000","last_change":"2024-03-19T03:06:27.529596+0000","last_active":"2024-03-19T03:06:27.529559+0000","last_peered":"2024-03-19T03:06:27.529559+0000","last_clean":"2024-03-19T03:06:27.529559+0000","last_became_active":"2024-03-19T02:54:56.213634+0000","last_became_peered":"2024-03-19T02:54:56.213634+0000","last_unstale":"2024-03-19T03:06:27.529559+0000","last_undegraded":"2024-03-19T03:06:27.529559+0000","last_fullsized":"2024-03-19T03:06:27.529559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'485","last_scrub_stamp":"2024-03-19T03:06:27.529493+0000","last_deep_scrub":"38'485","last_deep_scrub_stamp":"2024-03-19T03:06:27.529493+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.529493+0000","objects_scrubbed":164,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:45.129968+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0.068635079000000002,"stat_sum":{"num_bytes":22225330,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":36,"num_read_kb":12306,"num_write":348,"num_write_kb":21889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"38'489","reported_seq":2616,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:35.880844+0000","last_change":"2024-03-19T03:06:35.880872+0000","last_active":"2024-03-19T03:06:35.880844+0000","last_peered":"2024-03-19T03:06:35.880844+0000","last_clean":"2024-03-19T03:06:35.880844+0000","last_became_active":"2024-03-19T02:54:56.210511+0000","last_became_peered":"2024-03-19T02:54:56.210511+0000","last_unstale":"2024-03-19T03:06:35.880844+0000","last_undegraded":"2024-03-19T03:06:35.880844+0000","last_fullsized":"2024-03-19T03:06:35.880844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'489","last_scrub_stamp":"2024-03-19T03:06:35.880788+0000","last_deep_scrub":"38'489","last_deep_scrub_stamp":"2024-03-19T03:06:35.880788+0000","last_clean_scrub_stamp":"2024-03-19T03:06:35.880788+0000","objects_scrubbed":157,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:34.100521+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0.054050365000000003,"stat_sum":{"num_bytes":13922214,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":36,"num_read_kb":4118,"num_write":345,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"38'564","reported_seq":2688,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.671909+0000","last_change":"2024-03-19T03:06:26.671960+0000","last_active":"2024-03-19T03:06:26.671909+0000","last_peered":"2024-03-19T03:06:26.671909+0000","last_clean":"2024-03-19T03:06:26.671909+0000","last_became_active":"2024-03-19T02:55:45.950634+0000","last_became_peered":"2024-03-19T02:55:45.950634+0000","last_unstale":"2024-03-19T03:06:26.671909+0000","last_undegraded":"2024-03-19T03:06:26.671909+0000","last_fullsized":"2024-03-19T03:06:26.671909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'564","last_scrub_stamp":"2024-03-19T03:06:26.671833+0000","last_deep_scrub":"38'564","last_deep_scrub_stamp":"2024-03-19T03:06:26.671833+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.671833+0000","objects_scrubbed":196,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:34:04.330536+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0.050892488,"stat_sum":{"num_bytes":11658804,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":27,"num_read_kb":4113,"num_write":395,"num_write_kb":15629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"38'502","reported_seq":2629,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:32.707721+0000","last_change":"2024-03-19T03:06:32.707789+0000","last_active":"2024-03-19T03:06:32.707721+0000","last_peered":"2024-03-19T03:06:32.707721+0000","last_clean":"2024-03-19T03:06:32.707721+0000","last_became_active":"2024-03-19T02:54:56.209670+0000","last_became_peered":"2024-03-19T02:54:56.209670+0000","last_unstale":"2024-03-19T03:06:32.707721+0000","last_undegraded":"2024-03-19T03:06:32.707721+0000","last_fullsized":"2024-03-19T03:06:32.707721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'502","last_scrub_stamp":"2024-03-19T03:06:32.707618+0000","last_deep_scrub":"38'502","last_deep_scrub_stamp":"2024-03-19T03:06:32.707618+0000","last_clean_scrub_stamp":"2024-03-19T03:06:32.707618+0000","objects_scrubbed":167,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:35.683489+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0.035464137999999999,"stat_sum":{"num_bytes":11414841,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":36,"num_read_kb":4123,"num_write":359,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"38'472","reported_seq":2585,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.816605+0000","last_change":"2024-03-19T03:06:24.816666+0000","last_active":"2024-03-19T03:06:24.816605+0000","last_peered":"2024-03-19T03:06:24.816605+0000","last_clean":"2024-03-19T03:06:24.816605+0000","last_became_active":"2024-03-19T02:54:56.196752+0000","last_became_peered":"2024-03-19T02:54:56.196752+0000","last_unstale":"2024-03-19T03:06:24.816605+0000","last_undegraded":"2024-03-19T03:06:24.816605+0000","last_fullsized":"2024-03-19T03:06:24.816605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'472","last_scrub_stamp":"2024-03-19T03:06:24.816527+0000","last_deep_scrub":"38'472","last_deep_scrub_stamp":"2024-03-19T03:06:24.816527+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.816527+0000","objects_scrubbed":166,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:41.261124+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0.016612378000000001,"stat_sum":{"num_bytes":15052974,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":23,"num_read_kb":4112,"num_write":322,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"38'51","reported_seq":123,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.744256+0000","last_change":"2024-03-19T03:06:21.744305+0000","last_active":"2024-03-19T03:06:21.744256+0000","last_peered":"2024-03-19T03:06:21.744256+0000","last_clean":"2024-03-19T03:06:21.744256+0000","last_became_active":"2024-03-19T02:54:55.205564+0000","last_became_peered":"2024-03-19T02:54:55.205564+0000","last_unstale":"2024-03-19T03:06:21.744256+0000","last_undegraded":"2024-03-19T03:06:21.744256+0000","last_fullsized":"2024-03-19T03:06:21.744256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'51","last_scrub_stamp":"2024-03-19T03:06:21.744120+0000","last_deep_scrub":"38'51","last_deep_scrub_stamp":"2024-03-19T03:06:21.744120+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.744120+0000","objects_scrubbed":2,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:02:27.068022+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":52,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1200,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"38'423","reported_seq":2538,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.379542+0000","last_change":"2024-03-19T03:06:20.379618+0000","last_active":"2024-03-19T03:06:20.379542+0000","last_peered":"2024-03-19T03:06:20.379542+0000","last_clean":"2024-03-19T03:06:20.379542+0000","last_became_active":"2024-03-19T02:54:56.199890+0000","last_became_peered":"2024-03-19T02:54:56.199890+0000","last_unstale":"2024-03-19T03:06:20.379542+0000","last_undegraded":"2024-03-19T03:06:20.379542+0000","last_fullsized":"2024-03-19T03:06:20.379542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'423","last_scrub_stamp":"2024-03-19T03:06:20.379399+0000","last_deep_scrub":"38'423","last_deep_scrub_stamp":"2024-03-19T03:06:20.379399+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.379399+0000","objects_scrubbed":140,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:26.615221+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0.066278250999999996,"stat_sum":{"num_bytes":6674098,"num_objects":140,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":26,"num_read_kb":15,"num_write":306,"num_write_kb":6677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"38'204","reported_seq":283,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.556405+0000","last_change":"2024-03-19T03:06:24.556457+0000","last_active":"2024-03-19T03:06:24.556405+0000","last_peered":"2024-03-19T03:06:24.556405+0000","last_clean":"2024-03-19T03:06:24.556405+0000","last_became_active":"2024-03-19T02:54:55.183684+0000","last_became_peered":"2024-03-19T02:54:55.183684+0000","last_unstale":"2024-03-19T03:06:24.556405+0000","last_undegraded":"2024-03-19T03:06:24.556405+0000","last_fullsized":"2024-03-19T03:06:24.556405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'204","last_scrub_stamp":"2024-03-19T03:06:24.556299+0000","last_deep_scrub":"38'204","last_deep_scrub_stamp":"2024-03-19T03:06:24.556299+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.556299+0000","objects_scrubbed":6,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:48:16.014335+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":61440,"num_write":198,"num_write_kb":16593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5990,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"38'31","reported_seq":98,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.744372+0000","last_change":"2024-03-19T03:06:19.744508+0000","last_active":"2024-03-19T03:06:19.744372+0000","last_peered":"2024-03-19T03:06:19.744372+0000","last_clean":"2024-03-19T03:06:19.744372+0000","last_became_active":"2024-03-19T02:54:55.189882+0000","last_became_peered":"2024-03-19T02:54:55.189882+0000","last_unstale":"2024-03-19T03:06:19.744372+0000","last_undegraded":"2024-03-19T03:06:19.744372+0000","last_fullsized":"2024-03-19T03:06:19.744372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'31","last_scrub_stamp":"2024-03-19T03:06:19.744110+0000","last_deep_scrub":"38'31","last_deep_scrub_stamp":"2024-03-19T03:06:19.744110+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.744110+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:05.113905+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":29,"num_write_kb":4131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27583,"num_omap_keys":47,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"38'531","reported_seq":2656,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:30.088691+0000","last_change":"2024-03-19T03:06:30.088743+0000","last_active":"2024-03-19T03:06:30.088691+0000","last_peered":"2024-03-19T03:06:30.088691+0000","last_clean":"2024-03-19T03:06:30.088691+0000","last_became_active":"2024-03-19T02:54:56.205620+0000","last_became_peered":"2024-03-19T02:54:56.205620+0000","last_unstale":"2024-03-19T03:06:30.088691+0000","last_undegraded":"2024-03-19T03:06:30.088691+0000","last_fullsized":"2024-03-19T03:06:30.088691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'531","last_scrub_stamp":"2024-03-19T03:06:30.088520+0000","last_deep_scrub":"38'531","last_deep_scrub_stamp":"2024-03-19T03:06:30.088520+0000","last_clean_scrub_stamp":"2024-03-19T03:06:30.088520+0000","objects_scrubbed":176,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:22.284050+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0.061005943,"stat_sum":{"num_bytes":22416031,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":34,"num_read_kb":16407,"num_write":386,"num_write_kb":31278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"38'755","reported_seq":827,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.554513+0000","last_change":"2024-03-19T03:06:21.554548+0000","last_active":"2024-03-19T03:06:21.554513+0000","last_peered":"2024-03-19T03:06:21.554513+0000","last_clean":"2024-03-19T03:06:21.554513+0000","last_became_active":"2024-03-19T02:54:55.175267+0000","last_became_peered":"2024-03-19T02:54:55.175267+0000","last_unstale":"2024-03-19T03:06:21.554513+0000","last_undegraded":"2024-03-19T03:06:21.554513+0000","last_fullsized":"2024-03-19T03:06:21.554513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'755","last_scrub_stamp":"2024-03-19T03:06:21.554453+0000","last_deep_scrub":"38'755","last_deep_scrub_stamp":"2024-03-19T03:06:21.554453+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.554453+0000","objects_scrubbed":6,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:33.594671+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":748,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9712,"num_omap_keys":16,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"38'457","reported_seq":2577,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.509891+0000","last_change":"2024-03-19T03:06:27.509955+0000","last_active":"2024-03-19T03:06:27.509891+0000","last_peered":"2024-03-19T03:06:27.509891+0000","last_clean":"2024-03-19T03:06:27.509891+0000","last_became_active":"2024-03-19T02:54:56.212019+0000","last_became_peered":"2024-03-19T02:54:56.212019+0000","last_unstale":"2024-03-19T03:06:27.509891+0000","last_undegraded":"2024-03-19T03:06:27.509891+0000","last_fullsized":"2024-03-19T03:06:27.509891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'457","last_scrub_stamp":"2024-03-19T03:06:27.509794+0000","last_deep_scrub":"38'457","last_deep_scrub_stamp":"2024-03-19T03:06:27.509794+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.509794+0000","objects_scrubbed":160,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:56:11.421697+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0.048289845999999997,"stat_sum":{"num_bytes":20433584,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":31,"num_read_kb":16402,"num_write":319,"num_write_kb":24181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"38'11","reported_seq":74,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.486060+0000","last_change":"2024-03-19T03:06:22.486103+0000","last_active":"2024-03-19T03:06:22.486060+0000","last_peered":"2024-03-19T03:06:22.486060+0000","last_clean":"2024-03-19T03:06:22.486060+0000","last_became_active":"2024-03-19T02:54:55.182347+0000","last_became_peered":"2024-03-19T02:54:55.182347+0000","last_unstale":"2024-03-19T03:06:22.486060+0000","last_undegraded":"2024-03-19T03:06:22.486060+0000","last_fullsized":"2024-03-19T03:06:22.486060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'11","last_scrub_stamp":"2024-03-19T03:06:22.485993+0000","last_deep_scrub":"38'11","last_deep_scrub_stamp":"2024-03-19T03:06:22.485993+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.485993+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:34.035737+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33770,"num_omap_keys":56,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"38'490","reported_seq":2606,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.647759+0000","last_change":"2024-03-19T03:06:25.647860+0000","last_active":"2024-03-19T03:06:25.647759+0000","last_peered":"2024-03-19T03:06:25.647759+0000","last_clean":"2024-03-19T03:06:25.647759+0000","last_became_active":"2024-03-19T02:55:45.403591+0000","last_became_peered":"2024-03-19T02:55:45.403591+0000","last_unstale":"2024-03-19T03:06:25.647759+0000","last_undegraded":"2024-03-19T03:06:25.647759+0000","last_fullsized":"2024-03-19T03:06:25.647759+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'490","last_scrub_stamp":"2024-03-19T03:06:25.647637+0000","last_deep_scrub":"38'490","last_deep_scrub_stamp":"2024-03-19T03:06:25.647637+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.647637+0000","objects_scrubbed":160,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:33:48.408152+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0.050955334999999997,"stat_sum":{"num_bytes":22779940,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":32,"num_read_kb":10328,"num_write":348,"num_write_kb":22426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"38'1628","reported_seq":1697,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.596054+0000","last_change":"2024-03-19T03:06:23.596090+0000","last_active":"2024-03-19T03:06:23.596054+0000","last_peered":"2024-03-19T03:06:23.596054+0000","last_clean":"2024-03-19T03:06:23.596054+0000","last_became_active":"2024-03-19T02:54:55.182177+0000","last_became_peered":"2024-03-19T02:54:55.182177+0000","last_unstale":"2024-03-19T03:06:23.596054+0000","last_undegraded":"2024-03-19T03:06:23.596054+0000","last_fullsized":"2024-03-19T03:06:23.596054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'1628","last_scrub_stamp":"2024-03-19T03:06:23.595988+0000","last_deep_scrub":"38'1628","last_deep_scrub_stamp":"2024-03-19T03:06:23.595988+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.595988+0000","objects_scrubbed":9,"log_size":1628,"log_dups_size":0,"ondisk_log_size":1628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:34.710849+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":19565,"num_write":1617,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":44592,"num_omap_keys":76,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"38'432","reported_seq":2554,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.532600+0000","last_change":"2024-03-19T03:06:28.532656+0000","last_active":"2024-03-19T03:06:28.532600+0000","last_peered":"2024-03-19T03:06:28.532600+0000","last_clean":"2024-03-19T03:06:28.532600+0000","last_became_active":"2024-03-19T02:55:45.952615+0000","last_became_peered":"2024-03-19T02:55:45.952615+0000","last_unstale":"2024-03-19T03:06:28.532600+0000","last_undegraded":"2024-03-19T03:06:28.532600+0000","last_fullsized":"2024-03-19T03:06:28.532600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'432","last_scrub_stamp":"2024-03-19T03:06:28.532529+0000","last_deep_scrub":"38'432","last_deep_scrub_stamp":"2024-03-19T03:06:28.532529+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.532529+0000","objects_scrubbed":147,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:16.234577+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0.067870387000000004,"stat_sum":{"num_bytes":9678845,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":28,"num_read_kb":4114,"num_write":307,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"38'353","reported_seq":431,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:31.604931+0000","last_change":"2024-03-19T03:06:18.329442+0000","last_active":"2024-03-19T03:06:31.604931+0000","last_peered":"2024-03-19T03:06:31.604931+0000","last_clean":"2024-03-19T03:06:31.604931+0000","last_became_active":"2024-03-19T02:54:55.193100+0000","last_became_peered":"2024-03-19T02:54:55.193100+0000","last_unstale":"2024-03-19T03:06:31.604931+0000","last_undegraded":"2024-03-19T03:06:31.604931+0000","last_fullsized":"2024-03-19T03:06:31.604931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'335","last_scrub_stamp":"2024-03-19T03:06:18.329106+0000","last_deep_scrub":"38'335","last_deep_scrub_stamp":"2024-03-19T03:06:18.329106+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.329106+0000","objects_scrubbed":5,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:05.059377+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10583320,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":16407,"num_write":346,"num_write_kb":10528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"38'454","reported_seq":2568,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.771036+0000","last_change":"2024-03-19T03:06:22.771036+0000","last_active":"2024-03-19T03:06:22.771036+0000","last_peered":"2024-03-19T03:06:22.771036+0000","last_clean":"2024-03-19T03:06:22.771036+0000","last_became_active":"2024-03-19T02:54:56.190767+0000","last_became_peered":"2024-03-19T02:54:56.190767+0000","last_unstale":"2024-03-19T03:06:22.771036+0000","last_undegraded":"2024-03-19T03:06:22.771036+0000","last_fullsized":"2024-03-19T03:06:22.771036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'454","last_scrub_stamp":"2024-03-19T03:06:22.770929+0000","last_deep_scrub":"38'454","last_deep_scrub_stamp":"2024-03-19T03:06:22.770929+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.770929+0000","objects_scrubbed":158,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:45:13.792906+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.016376714000000001,"stat_sum":{"num_bytes":15448536,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":23,"num_read_kb":4110,"num_write":308,"num_write_kb":19342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"38'169","reported_seq":263,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:35.925934+0000","last_change":"2024-03-19T03:06:25.581611+0000","last_active":"2024-03-19T03:06:35.925934+0000","last_peered":"2024-03-19T03:06:35.925934+0000","last_clean":"2024-03-19T03:06:35.925934+0000","last_became_active":"2024-03-19T02:54:55.191688+0000","last_became_peered":"2024-03-19T02:54:55.191688+0000","last_unstale":"2024-03-19T03:06:35.925934+0000","last_undegraded":"2024-03-19T03:06:35.925934+0000","last_fullsized":"2024-03-19T03:06:35.925934+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'168","last_scrub_stamp":"2024-03-19T03:06:25.581522+0000","last_deep_scrub":"38'168","last_deep_scrub_stamp":"2024-03-19T03:06:25.581522+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.581522+0000","objects_scrubbed":7,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:45.561056+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":57372,"num_write":156,"num_write_kb":12389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"38'490","reported_seq":2607,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:39.914950+0000","last_change":"2024-03-19T03:06:39.915009+0000","last_active":"2024-03-19T03:06:39.914950+0000","last_peered":"2024-03-19T03:06:39.914950+0000","last_clean":"2024-03-19T03:06:39.914950+0000","last_became_active":"2024-03-19T02:54:56.211403+0000","last_became_peered":"2024-03-19T02:54:56.211403+0000","last_unstale":"2024-03-19T03:06:39.914950+0000","last_undegraded":"2024-03-19T03:06:39.914950+0000","last_fullsized":"2024-03-19T03:06:39.914950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'490","last_scrub_stamp":"2024-03-19T03:06:39.914888+0000","last_deep_scrub":"38'490","last_deep_scrub_stamp":"2024-03-19T03:06:39.914888+0000","last_clean_scrub_stamp":"2024-03-19T03:06:39.914888+0000","objects_scrubbed":167,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:57:30.627423+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0.053512995000000001,"stat_sum":{"num_bytes":2174562,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":27,"num_read_kb":19,"num_write":323,"num_write_kb":2244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"38'159","reported_seq":243,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:17.216450+0000","last_change":"2024-03-19T03:06:17.216450+0000","last_active":"2024-03-19T03:06:17.216450+0000","last_peered":"2024-03-19T03:06:17.216450+0000","last_clean":"2024-03-19T03:06:17.216450+0000","last_became_active":"2024-03-19T02:54:55.197858+0000","last_became_peered":"2024-03-19T02:54:55.197858+0000","last_unstale":"2024-03-19T03:06:17.216450+0000","last_undegraded":"2024-03-19T03:06:17.216450+0000","last_fullsized":"2024-03-19T03:06:17.216450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'159","last_scrub_stamp":"2024-03-19T03:06:17.216259+0000","last_deep_scrub":"38'159","last_deep_scrub_stamp":"2024-03-19T03:06:17.216259+0000","last_clean_scrub_stamp":"2024-03-19T03:06:17.216259+0000","objects_scrubbed":5,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:49.902547+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":41000,"num_write":152,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14584,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"38'480","reported_seq":2609,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.330986+0000","last_change":"2024-03-19T03:06:22.330986+0000","last_active":"2024-03-19T03:06:22.330986+0000","last_peered":"2024-03-19T03:06:22.330986+0000","last_clean":"2024-03-19T03:06:22.330986+0000","last_became_active":"2024-03-19T02:54:56.207304+0000","last_became_peered":"2024-03-19T02:54:56.207304+0000","last_unstale":"2024-03-19T03:06:22.330986+0000","last_undegraded":"2024-03-19T03:06:22.330986+0000","last_fullsized":"2024-03-19T03:06:22.330986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'480","last_scrub_stamp":"2024-03-19T03:06:22.330860+0000","last_deep_scrub":"38'480","last_deep_scrub_stamp":"2024-03-19T03:06:22.330860+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.330860+0000","objects_scrubbed":177,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:39.319271+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0.068156319000000007,"stat_sum":{"num_bytes":18375725,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":37,"num_read_kb":16403,"num_write":329,"num_write_kb":26314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"38'2","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.553582+0000","last_change":"2024-03-19T03:06:24.553582+0000","last_active":"2024-03-19T03:06:24.553582+0000","last_peered":"2024-03-19T03:06:24.553582+0000","last_clean":"2024-03-19T03:06:24.553582+0000","last_became_active":"2024-03-19T02:54:55.199530+0000","last_became_peered":"2024-03-19T02:54:55.199530+0000","last_unstale":"2024-03-19T03:06:24.553582+0000","last_undegraded":"2024-03-19T03:06:24.553582+0000","last_fullsized":"2024-03-19T03:06:24.553582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'2","last_scrub_stamp":"2024-03-19T03:06:24.553443+0000","last_deep_scrub":"38'2","last_deep_scrub_stamp":"2024-03-19T03:06:24.553443+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.553443+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:56:20.435689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15574,"num_omap_keys":26,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"38'488","reported_seq":2609,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.988527+0000","last_change":"2024-03-19T03:06:23.988527+0000","last_active":"2024-03-19T03:06:23.988527+0000","last_peered":"2024-03-19T03:06:23.988527+0000","last_clean":"2024-03-19T03:06:23.988527+0000","last_became_active":"2024-03-19T02:54:56.194201+0000","last_became_peered":"2024-03-19T02:54:56.194201+0000","last_unstale":"2024-03-19T03:06:23.988527+0000","last_undegraded":"2024-03-19T03:06:23.988527+0000","last_fullsized":"2024-03-19T03:06:23.988527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'488","last_scrub_stamp":"2024-03-19T03:06:23.988386+0000","last_deep_scrub":"38'488","last_deep_scrub_stamp":"2024-03-19T03:06:23.988386+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.988386+0000","objects_scrubbed":167,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:18.163193+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0.051813761,"stat_sum":{"num_bytes":7359660,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":339,"num_write_kb":7364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"38'2","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.516316+0000","last_change":"2024-03-19T03:06:23.516316+0000","last_active":"2024-03-19T03:06:23.516316+0000","last_peered":"2024-03-19T03:06:23.516316+0000","last_clean":"2024-03-19T03:06:23.516316+0000","last_became_active":"2024-03-19T02:54:55.183024+0000","last_became_peered":"2024-03-19T02:54:55.183024+0000","last_unstale":"2024-03-19T03:06:23.516316+0000","last_undegraded":"2024-03-19T03:06:23.516316+0000","last_fullsized":"2024-03-19T03:06:23.516316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'2","last_scrub_stamp":"2024-03-19T03:06:23.516227+0000","last_deep_scrub":"38'2","last_deep_scrub_stamp":"2024-03-19T03:06:23.516227+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.516227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:28.501997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11381,"num_omap_keys":19,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"38'495","reported_seq":2622,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:31.780644+0000","last_change":"2024-03-19T03:06:31.780644+0000","last_active":"2024-03-19T03:06:31.780644+0000","last_peered":"2024-03-19T03:06:31.780644+0000","last_clean":"2024-03-19T03:06:31.780644+0000","last_became_active":"2024-03-19T02:54:56.198806+0000","last_became_peered":"2024-03-19T02:54:56.198806+0000","last_unstale":"2024-03-19T03:06:31.780644+0000","last_undegraded":"2024-03-19T03:06:31.780644+0000","last_fullsized":"2024-03-19T03:06:31.780644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'495","last_scrub_stamp":"2024-03-19T03:06:31.780457+0000","last_deep_scrub":"38'495","last_deep_scrub_stamp":"2024-03-19T03:06:31.780457+0000","last_clean_scrub_stamp":"2024-03-19T03:06:31.780457+0000","objects_scrubbed":173,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:15.503870+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0.035899844,"stat_sum":{"num_bytes":22036853,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":37,"num_read_kb":12302,"num_write":335,"num_write_kb":21697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"38'137","reported_seq":234,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.606482+0000","last_change":"2024-03-19T03:06:21.606482+0000","last_active":"2024-03-19T03:06:21.606482+0000","last_peered":"2024-03-19T03:06:21.606482+0000","last_clean":"2024-03-19T03:06:21.606482+0000","last_became_active":"2024-03-19T02:54:55.198916+0000","last_became_peered":"2024-03-19T02:54:55.198916+0000","last_unstale":"2024-03-19T03:06:21.606482+0000","last_undegraded":"2024-03-19T03:06:21.606482+0000","last_fullsized":"2024-03-19T03:06:21.606482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'137","last_scrub_stamp":"2024-03-19T03:06:21.606300+0000","last_deep_scrub":"38'137","last_deep_scrub_stamp":"2024-03-19T03:06:21.606300+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.606300+0000","objects_scrubbed":10,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:50:17.981331+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":31075,"num_write":133,"num_write_kb":8289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33159,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"38'517","reported_seq":2642,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:32.933218+0000","last_change":"2024-03-19T03:06:32.933218+0000","last_active":"2024-03-19T03:06:32.933218+0000","last_peered":"2024-03-19T03:06:32.933218+0000","last_clean":"2024-03-19T03:06:32.933218+0000","last_became_active":"2024-03-19T02:54:56.195042+0000","last_became_peered":"2024-03-19T02:54:56.195042+0000","last_unstale":"2024-03-19T03:06:32.933218+0000","last_undegraded":"2024-03-19T03:06:32.933218+0000","last_fullsized":"2024-03-19T03:06:32.933218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'517","last_scrub_stamp":"2024-03-19T03:06:32.933123+0000","last_deep_scrub":"38'517","last_deep_scrub_stamp":"2024-03-19T03:06:32.933123+0000","last_clean_scrub_stamp":"2024-03-19T03:06:32.933123+0000","objects_scrubbed":183,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:34:32.644326+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0.051911553999999999,"stat_sum":{"num_bytes":11598304,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":33,"num_read_kb":12305,"num_write":357,"num_write_kb":19699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"38'37","reported_seq":110,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.601771+0000","last_change":"2024-03-19T03:06:20.601820+0000","last_active":"2024-03-19T03:06:20.601771+0000","last_peered":"2024-03-19T03:06:20.601771+0000","last_clean":"2024-03-19T03:06:20.601771+0000","last_became_active":"2024-03-19T02:54:55.197995+0000","last_became_peered":"2024-03-19T02:54:55.197995+0000","last_unstale":"2024-03-19T03:06:20.601771+0000","last_undegraded":"2024-03-19T03:06:20.601771+0000","last_fullsized":"2024-03-19T03:06:20.601771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'37","last_scrub_stamp":"2024-03-19T03:06:20.601707+0000","last_deep_scrub":"38'37","last_deep_scrub_stamp":"2024-03-19T03:06:20.601707+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.601707+0000","objects_scrubbed":9,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:51.991051+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":35,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75111,"num_omap_keys":125,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"38'436","reported_seq":2545,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:38.852985+0000","last_change":"2024-03-19T03:06:38.853028+0000","last_active":"2024-03-19T03:06:38.852985+0000","last_peered":"2024-03-19T03:06:38.852985+0000","last_clean":"2024-03-19T03:06:38.852985+0000","last_became_active":"2024-03-19T02:54:56.202038+0000","last_became_peered":"2024-03-19T02:54:56.202038+0000","last_unstale":"2024-03-19T03:06:38.852985+0000","last_undegraded":"2024-03-19T03:06:38.852985+0000","last_fullsized":"2024-03-19T03:06:38.852985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'436","last_scrub_stamp":"2024-03-19T03:06:38.852920+0000","last_deep_scrub":"38'436","last_deep_scrub_stamp":"2024-03-19T03:06:38.852920+0000","last_clean_scrub_stamp":"2024-03-19T03:06:38.852920+0000","objects_scrubbed":172,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:34:35.782090+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0.052296778000000002,"stat_sum":{"num_bytes":10391613,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":18,"num_read_kb":6155,"num_write":296,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"38'119","reported_seq":193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.575116+0000","last_change":"2024-03-19T03:06:19.575159+0000","last_active":"2024-03-19T03:06:19.575116+0000","last_peered":"2024-03-19T03:06:19.575116+0000","last_clean":"2024-03-19T03:06:19.575116+0000","last_became_active":"2024-03-19T02:54:55.191774+0000","last_became_peered":"2024-03-19T02:54:55.191774+0000","last_unstale":"2024-03-19T03:06:19.575116+0000","last_undegraded":"2024-03-19T03:06:19.575116+0000","last_fullsized":"2024-03-19T03:06:19.575116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'119","last_scrub_stamp":"2024-03-19T03:06:19.575016+0000","last_deep_scrub":"38'119","last_deep_scrub_stamp":"2024-03-19T03:06:19.575016+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.575016+0000","objects_scrubbed":3,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:16.168974+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":116,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9655,"num_omap_keys":17,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"38'459","reported_seq":2578,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:33.957840+0000","last_change":"2024-03-19T03:06:33.957873+0000","last_active":"2024-03-19T03:06:33.957840+0000","last_peered":"2024-03-19T03:06:33.957840+0000","last_clean":"2024-03-19T03:06:33.957840+0000","last_became_active":"2024-03-19T02:54:56.196658+0000","last_became_peered":"2024-03-19T02:54:56.196658+0000","last_unstale":"2024-03-19T03:06:33.957840+0000","last_undegraded":"2024-03-19T03:06:33.957840+0000","last_fullsized":"2024-03-19T03:06:33.957840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'459","last_scrub_stamp":"2024-03-19T03:06:33.957780+0000","last_deep_scrub":"38'459","last_deep_scrub_stamp":"2024-03-19T03:06:33.957780+0000","last_clean_scrub_stamp":"2024-03-19T03:06:33.957780+0000","objects_scrubbed":163,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:21:40.997825+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0.044895387000000002,"stat_sum":{"num_bytes":10001648,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":28,"num_read_kb":8211,"num_write":321,"num_write_kb":14027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"38'22","reported_seq":94,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.342109+0000","last_change":"2024-03-19T03:06:20.342137+0000","last_active":"2024-03-19T03:06:20.342109+0000","last_peered":"2024-03-19T03:06:20.342109+0000","last_clean":"2024-03-19T03:06:20.342109+0000","last_became_active":"2024-03-19T02:54:55.198025+0000","last_became_peered":"2024-03-19T02:54:55.198025+0000","last_unstale":"2024-03-19T03:06:20.342109+0000","last_undegraded":"2024-03-19T03:06:20.342109+0000","last_fullsized":"2024-03-19T03:06:20.342109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'22","last_scrub_stamp":"2024-03-19T03:06:20.341995+0000","last_deep_scrub":"38'22","last_deep_scrub_stamp":"2024-03-19T03:06:20.341995+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.341995+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:15.931474+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":20,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8348,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"38'518","reported_seq":2632,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:30.824965+0000","last_change":"2024-03-19T03:06:30.824988+0000","last_active":"2024-03-19T03:06:30.824965+0000","last_peered":"2024-03-19T03:06:30.824965+0000","last_clean":"2024-03-19T03:06:30.824965+0000","last_became_active":"2024-03-19T02:54:56.194783+0000","last_became_peered":"2024-03-19T02:54:56.194783+0000","last_unstale":"2024-03-19T03:06:30.824965+0000","last_undegraded":"2024-03-19T03:06:30.824965+0000","last_fullsized":"2024-03-19T03:06:30.824965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'518","last_scrub_stamp":"2024-03-19T03:06:30.824905+0000","last_deep_scrub":"38'518","last_deep_scrub_stamp":"2024-03-19T03:06:30.824905+0000","last_clean_scrub_stamp":"2024-03-19T03:06:30.824905+0000","objects_scrubbed":178,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:33.633212+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0.055101615,"stat_sum":{"num_bytes":11152492,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":22,"num_read_kb":4110,"num_write":358,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"38'108","reported_seq":173,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:18.615640+0000","last_change":"2024-03-19T03:06:18.615729+0000","last_active":"2024-03-19T03:06:18.615640+0000","last_peered":"2024-03-19T03:06:18.615640+0000","last_clean":"2024-03-19T03:06:18.615640+0000","last_became_active":"2024-03-19T02:54:55.189121+0000","last_became_peered":"2024-03-19T02:54:55.189121+0000","last_unstale":"2024-03-19T03:06:18.615640+0000","last_undegraded":"2024-03-19T03:06:18.615640+0000","last_fullsized":"2024-03-19T03:06:18.615640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'108","last_scrub_stamp":"2024-03-19T03:06:18.615428+0000","last_deep_scrub":"38'108","last_deep_scrub_stamp":"2024-03-19T03:06:18.615428+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.615428+0000","objects_scrubbed":3,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:33.816723+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":108,"num_write_kb":4216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10273,"num_omap_keys":17,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'470","reported_seq":2591,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:37.898535+0000","last_change":"2024-03-19T03:06:37.898562+0000","last_active":"2024-03-19T03:06:37.898535+0000","last_peered":"2024-03-19T03:06:37.898535+0000","last_clean":"2024-03-19T03:06:37.898535+0000","last_became_active":"2024-03-19T02:54:56.202274+0000","last_became_peered":"2024-03-19T02:54:56.202274+0000","last_unstale":"2024-03-19T03:06:37.898535+0000","last_undegraded":"2024-03-19T03:06:37.898535+0000","last_fullsized":"2024-03-19T03:06:37.898535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'470","last_scrub_stamp":"2024-03-19T03:06:37.898474+0000","last_deep_scrub":"38'470","last_deep_scrub_stamp":"2024-03-19T03:06:37.898474+0000","last_clean_scrub_stamp":"2024-03-19T03:06:37.898474+0000","objects_scrubbed":156,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:32.458642+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0.051414362999999998,"stat_sum":{"num_bytes":15817335,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":30,"num_read_kb":4113,"num_write":329,"num_write_kb":19678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"38'119","reported_seq":222,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.813666+0000","last_change":"2024-03-19T03:06:24.813696+0000","last_active":"2024-03-19T03:06:24.813666+0000","last_peered":"2024-03-19T03:06:24.813666+0000","last_clean":"2024-03-19T03:06:24.813666+0000","last_became_active":"2024-03-19T02:54:52.041775+0000","last_became_peered":"2024-03-19T02:54:52.041775+0000","last_unstale":"2024-03-19T03:06:24.813666+0000","last_undegraded":"2024-03-19T03:06:24.813666+0000","last_fullsized":"2024-03-19T03:06:24.813666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'119","last_scrub_stamp":"2024-03-19T03:06:24.813591+0000","last_deep_scrub":"38'119","last_deep_scrub_stamp":"2024-03-19T03:06:24.813591+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.813591+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:16:27.589427+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"38'1699","reported_seq":1766,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.539453+0000","last_change":"2024-03-19T03:06:22.539479+0000","last_active":"2024-03-19T03:06:22.539453+0000","last_peered":"2024-03-19T03:06:22.539453+0000","last_clean":"2024-03-19T03:06:22.539453+0000","last_became_active":"2024-03-19T02:54:55.182719+0000","last_became_peered":"2024-03-19T02:54:55.182719+0000","last_unstale":"2024-03-19T03:06:22.539453+0000","last_undegraded":"2024-03-19T03:06:22.539453+0000","last_fullsized":"2024-03-19T03:06:22.539453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'1699","last_scrub_stamp":"2024-03-19T03:06:22.539385+0000","last_deep_scrub":"38'1699","last_deep_scrub_stamp":"2024-03-19T03:06:22.539385+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.539385+0000","objects_scrubbed":2,"log_size":1699,"log_dups_size":0,"ondisk_log_size":1699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:22:25.129408+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5356,"num_write":1694,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9318,"num_omap_keys":16,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"38'459","reported_seq":2575,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:29.044005+0000","last_change":"2024-03-19T03:06:29.044086+0000","last_active":"2024-03-19T03:06:29.044005+0000","last_peered":"2024-03-19T03:06:29.044005+0000","last_clean":"2024-03-19T03:06:29.044005+0000","last_became_active":"2024-03-19T02:54:56.204694+0000","last_became_peered":"2024-03-19T02:54:56.204694+0000","last_unstale":"2024-03-19T03:06:29.044005+0000","last_undegraded":"2024-03-19T03:06:29.044005+0000","last_fullsized":"2024-03-19T03:06:29.044005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'459","last_scrub_stamp":"2024-03-19T03:06:29.043825+0000","last_deep_scrub":"38'459","last_deep_scrub_stamp":"2024-03-19T03:06:29.043825+0000","last_clean_scrub_stamp":"2024-03-19T03:06:29.043825+0000","objects_scrubbed":162,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:04.750747+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0.063353052000000007,"stat_sum":{"num_bytes":11364310,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4110,"num_write":322,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"38'191","reported_seq":266,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.850405+0000","last_change":"2024-03-19T03:06:28.850434+0000","last_active":"2024-03-19T03:06:28.850405+0000","last_peered":"2024-03-19T03:06:28.850405+0000","last_clean":"2024-03-19T03:06:28.850405+0000","last_became_active":"2024-03-19T02:54:55.198138+0000","last_became_peered":"2024-03-19T02:54:55.198138+0000","last_unstale":"2024-03-19T03:06:28.850405+0000","last_undegraded":"2024-03-19T03:06:28.850405+0000","last_fullsized":"2024-03-19T03:06:28.850405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'191","last_scrub_stamp":"2024-03-19T03:06:28.850327+0000","last_deep_scrub":"38'191","last_deep_scrub_stamp":"2024-03-19T03:06:28.850327+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.850327+0000","objects_scrubbed":4,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:57.240679+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":181,"num_write_kb":12402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9013,"num_omap_keys":15,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"38'446","reported_seq":2557,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.384015+0000","last_change":"2024-03-19T03:06:19.384052+0000","last_active":"2024-03-19T03:06:19.384015+0000","last_peered":"2024-03-19T03:06:19.384015+0000","last_clean":"2024-03-19T03:06:19.384015+0000","last_became_active":"2024-03-19T02:54:56.198207+0000","last_became_peered":"2024-03-19T02:54:56.198207+0000","last_unstale":"2024-03-19T03:06:19.384015+0000","last_undegraded":"2024-03-19T03:06:19.384015+0000","last_fullsized":"2024-03-19T03:06:19.384015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'446","last_scrub_stamp":"2024-03-19T03:06:19.383952+0000","last_deep_scrub":"38'446","last_deep_scrub_stamp":"2024-03-19T03:06:19.383952+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.383952+0000","objects_scrubbed":163,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:49.812214+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0.068696631999999994,"stat_sum":{"num_bytes":14176427,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":313,"num_write_kb":14010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"38'95","reported_seq":252,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.582209+0000","last_change":"2024-03-19T03:06:20.582269+0000","last_active":"2024-03-19T03:06:20.582209+0000","last_peered":"2024-03-19T03:06:20.582209+0000","last_clean":"2024-03-19T03:06:20.582209+0000","last_became_active":"2024-03-19T02:54:55.194114+0000","last_became_peered":"2024-03-19T02:54:55.194114+0000","last_unstale":"2024-03-19T03:06:20.582209+0000","last_undegraded":"2024-03-19T03:06:20.582209+0000","last_fullsized":"2024-03-19T03:06:20.582209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'95","last_scrub_stamp":"2024-03-19T03:06:20.582062+0000","last_deep_scrub":"38'95","last_deep_scrub_stamp":"2024-03-19T03:06:20.582062+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.582062+0000","objects_scrubbed":2,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:25.216983+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":581,"num_write":90,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9584,"num_omap_keys":16,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"38'478","reported_seq":2595,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.504941+0000","last_change":"2024-03-19T03:06:26.504998+0000","last_active":"2024-03-19T03:06:26.504941+0000","last_peered":"2024-03-19T03:06:26.504941+0000","last_clean":"2024-03-19T03:06:26.504941+0000","last_became_active":"2024-03-19T02:54:56.207084+0000","last_became_peered":"2024-03-19T02:54:56.207084+0000","last_unstale":"2024-03-19T03:06:26.504941+0000","last_undegraded":"2024-03-19T03:06:26.504941+0000","last_fullsized":"2024-03-19T03:06:26.504941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'478","last_scrub_stamp":"2024-03-19T03:06:26.504838+0000","last_deep_scrub":"38'478","last_deep_scrub_stamp":"2024-03-19T03:06:26.504838+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.504838+0000","objects_scrubbed":174,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:56:01.596035+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0.044152405999999998,"stat_sum":{"num_bytes":9810356,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":324,"num_write_kb":9748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"38'64","reported_seq":137,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.606841+0000","last_change":"2024-03-19T03:06:25.606902+0000","last_active":"2024-03-19T03:06:25.606841+0000","last_peered":"2024-03-19T03:06:25.606841+0000","last_clean":"2024-03-19T03:06:25.606841+0000","last_became_active":"2024-03-19T02:54:55.200058+0000","last_became_peered":"2024-03-19T02:54:55.200058+0000","last_unstale":"2024-03-19T03:06:25.606841+0000","last_undegraded":"2024-03-19T03:06:25.606841+0000","last_fullsized":"2024-03-19T03:06:25.606841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'64","last_scrub_stamp":"2024-03-19T03:06:25.606765+0000","last_deep_scrub":"38'64","last_deep_scrub_stamp":"2024-03-19T03:06:25.606765+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.606765+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:18.908272+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":60,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"38'513","reported_seq":2632,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.851414+0000","last_change":"2024-03-19T03:06:25.851469+0000","last_active":"2024-03-19T03:06:25.851414+0000","last_peered":"2024-03-19T03:06:25.851414+0000","last_clean":"2024-03-19T03:06:25.851414+0000","last_became_active":"2024-03-19T02:54:56.215645+0000","last_became_peered":"2024-03-19T02:54:56.215645+0000","last_unstale":"2024-03-19T03:06:25.851414+0000","last_undegraded":"2024-03-19T03:06:25.851414+0000","last_fullsized":"2024-03-19T03:06:25.851414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'513","last_scrub_stamp":"2024-03-19T03:06:25.851361+0000","last_deep_scrub":"38'513","last_deep_scrub_stamp":"2024-03-19T03:06:25.851361+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.851361+0000","objects_scrubbed":174,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:26.594094+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0.016334565999999998,"stat_sum":{"num_bytes":7144435,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":28,"num_read_kb":4112,"num_write":358,"num_write_kb":7164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"38'364","reported_seq":428,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:22.306967+0000","last_change":"2024-03-19T03:06:22.307015+0000","last_active":"2024-03-19T03:06:22.306967+0000","last_peered":"2024-03-19T03:06:22.306967+0000","last_clean":"2024-03-19T03:06:22.306967+0000","last_became_active":"2024-03-19T02:54:55.204134+0000","last_became_peered":"2024-03-19T02:54:55.204134+0000","last_unstale":"2024-03-19T03:06:22.306967+0000","last_undegraded":"2024-03-19T03:06:22.306967+0000","last_fullsized":"2024-03-19T03:06:22.306967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'364","last_scrub_stamp":"2024-03-19T03:06:22.306855+0000","last_deep_scrub":"38'364","last_deep_scrub_stamp":"2024-03-19T03:06:22.306855+0000","last_clean_scrub_stamp":"2024-03-19T03:06:22.306855+0000","objects_scrubbed":1,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:52.447142+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":360,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'492","reported_seq":2612,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:34.854762+0000","last_change":"2024-03-19T03:06:34.854812+0000","last_active":"2024-03-19T03:06:34.854762+0000","last_peered":"2024-03-19T03:06:34.854762+0000","last_clean":"2024-03-19T03:06:34.854762+0000","last_became_active":"2024-03-19T02:54:56.203800+0000","last_became_peered":"2024-03-19T02:54:56.203800+0000","last_unstale":"2024-03-19T03:06:34.854762+0000","last_undegraded":"2024-03-19T03:06:34.854762+0000","last_fullsized":"2024-03-19T03:06:34.854762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'492","last_scrub_stamp":"2024-03-19T03:06:34.854659+0000","last_deep_scrub":"38'492","last_deep_scrub_stamp":"2024-03-19T03:06:34.854659+0000","last_clean_scrub_stamp":"2024-03-19T03:06:34.854659+0000","objects_scrubbed":179,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:54:41.148197+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0.054510613999999999,"stat_sum":{"num_bytes":18185428,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":28,"num_read_kb":9793,"num_write":337,"num_write_kb":22033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"38'366","reported_seq":435,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:30.814217+0000","last_change":"2024-03-19T03:06:30.814243+0000","last_active":"2024-03-19T03:06:30.814217+0000","last_peered":"2024-03-19T03:06:30.814217+0000","last_clean":"2024-03-19T03:06:30.814217+0000","last_became_active":"2024-03-19T02:54:55.204078+0000","last_became_peered":"2024-03-19T02:54:55.204078+0000","last_unstale":"2024-03-19T03:06:30.814217+0000","last_undegraded":"2024-03-19T03:06:30.814217+0000","last_fullsized":"2024-03-19T03:06:30.814217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'366","last_scrub_stamp":"2024-03-19T03:06:30.814155+0000","last_deep_scrub":"38'366","last_deep_scrub_stamp":"2024-03-19T03:06:30.814155+0000","last_clean_scrub_stamp":"2024-03-19T03:06:30.814155+0000","objects_scrubbed":2,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:51.822889+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":360,"num_write_kb":8331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"38'504","reported_seq":2622,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.492447+0000","last_change":"2024-03-19T03:06:25.492499+0000","last_active":"2024-03-19T03:06:25.492447+0000","last_peered":"2024-03-19T03:06:25.492447+0000","last_clean":"2024-03-19T03:06:25.492447+0000","last_became_active":"2024-03-19T02:54:56.206241+0000","last_became_peered":"2024-03-19T02:54:56.206241+0000","last_unstale":"2024-03-19T03:06:25.492447+0000","last_undegraded":"2024-03-19T03:06:25.492447+0000","last_fullsized":"2024-03-19T03:06:25.492447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'504","last_scrub_stamp":"2024-03-19T03:06:25.492373+0000","last_deep_scrub":"38'504","last_deep_scrub_stamp":"2024-03-19T03:06:25.492373+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.492373+0000","objects_scrubbed":181,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:40.276508+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0.047353098000000003,"stat_sum":{"num_bytes":7011110,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":27,"num_read_kb":4110,"num_write":348,"num_write_kb":6996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'272","reported_seq":349,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:19.342586+0000","last_change":"2024-03-19T03:06:19.342638+0000","last_active":"2024-03-19T03:06:19.342586+0000","last_peered":"2024-03-19T03:06:19.342586+0000","last_clean":"2024-03-19T03:06:19.342586+0000","last_became_active":"2024-03-19T02:54:55.195884+0000","last_became_peered":"2024-03-19T02:54:55.195884+0000","last_unstale":"2024-03-19T03:06:19.342586+0000","last_undegraded":"2024-03-19T03:06:19.342586+0000","last_fullsized":"2024-03-19T03:06:19.342586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'272","last_scrub_stamp":"2024-03-19T03:06:19.342467+0000","last_deep_scrub":"38'272","last_deep_scrub_stamp":"2024-03-19T03:06:19.342467+0000","last_clean_scrub_stamp":"2024-03-19T03:06:19.342467+0000","objects_scrubbed":2,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:20:18.043251+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":638,"num_write":618,"num_write_kb":2118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6878,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"38'421","reported_seq":2529,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:24.751095+0000","last_change":"2024-03-19T03:06:24.751095+0000","last_active":"2024-03-19T03:06:24.751095+0000","last_peered":"2024-03-19T03:06:24.751095+0000","last_clean":"2024-03-19T03:06:24.751095+0000","last_became_active":"2024-03-19T02:55:45.402992+0000","last_became_peered":"2024-03-19T02:55:45.402992+0000","last_unstale":"2024-03-19T03:06:24.751095+0000","last_undegraded":"2024-03-19T03:06:24.751095+0000","last_fullsized":"2024-03-19T03:06:24.751095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'421","last_scrub_stamp":"2024-03-19T03:06:24.750894+0000","last_deep_scrub":"38'421","last_deep_scrub_stamp":"2024-03-19T03:06:24.750894+0000","last_clean_scrub_stamp":"2024-03-19T03:06:24.750894+0000","objects_scrubbed":141,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:17:21.306833+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0.050031514999999999,"stat_sum":{"num_bytes":18622985,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":24,"num_read_kb":16,"num_write":307,"num_write_kb":22442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"38'152","reported_seq":226,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.546081+0000","last_change":"2024-03-19T03:06:21.546137+0000","last_active":"2024-03-19T03:06:21.546081+0000","last_peered":"2024-03-19T03:06:21.546081+0000","last_clean":"2024-03-19T03:06:21.546081+0000","last_became_active":"2024-03-19T02:54:55.180845+0000","last_became_peered":"2024-03-19T02:54:55.180845+0000","last_unstale":"2024-03-19T03:06:21.546081+0000","last_undegraded":"2024-03-19T03:06:21.546081+0000","last_fullsized":"2024-03-19T03:06:21.546081+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'152","last_scrub_stamp":"2024-03-19T03:06:21.545968+0000","last_deep_scrub":"38'152","last_deep_scrub_stamp":"2024-03-19T03:06:21.545968+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.545968+0000","objects_scrubbed":5,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:31:24.754946+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":9370,"num_write":147,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20965,"num_omap_keys":35,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"38'511","reported_seq":2642,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.485868+0000","last_change":"2024-03-19T03:06:23.485902+0000","last_active":"2024-03-19T03:06:23.485868+0000","last_peered":"2024-03-19T03:06:23.485868+0000","last_clean":"2024-03-19T03:06:23.485868+0000","last_became_active":"2024-03-19T02:54:56.209515+0000","last_became_peered":"2024-03-19T02:54:56.209515+0000","last_unstale":"2024-03-19T03:06:23.485868+0000","last_undegraded":"2024-03-19T03:06:23.485868+0000","last_fullsized":"2024-03-19T03:06:23.485868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'511","last_scrub_stamp":"2024-03-19T03:06:23.485785+0000","last_deep_scrub":"38'511","last_deep_scrub_stamp":"2024-03-19T03:06:23.485785+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.485785+0000","objects_scrubbed":178,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:21.300714+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0.068525034999999998,"stat_sum":{"num_bytes":35227376,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":40,"num_read_kb":16714,"num_write":347,"num_write_kb":34590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'20","reported_seq":88,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:29.845329+0000","last_change":"2024-03-19T03:06:29.845375+0000","last_active":"2024-03-19T03:06:29.845329+0000","last_peered":"2024-03-19T03:06:29.845329+0000","last_clean":"2024-03-19T03:06:29.845329+0000","last_became_active":"2024-03-19T02:54:55.198780+0000","last_became_peered":"2024-03-19T02:54:55.198780+0000","last_unstale":"2024-03-19T03:06:29.845329+0000","last_undegraded":"2024-03-19T03:06:29.845329+0000","last_fullsized":"2024-03-19T03:06:29.845329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'20","last_scrub_stamp":"2024-03-19T03:06:29.845240+0000","last_deep_scrub":"38'20","last_deep_scrub_stamp":"2024-03-19T03:06:29.845240+0000","last_clean_scrub_stamp":"2024-03-19T03:06:29.845240+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:32.539997+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":20,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":47848,"num_omap_keys":81,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"38'479","reported_seq":2600,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.856248+0000","last_change":"2024-03-19T03:06:23.856286+0000","last_active":"2024-03-19T03:06:23.856248+0000","last_peered":"2024-03-19T03:06:23.856248+0000","last_clean":"2024-03-19T03:06:23.856248+0000","last_became_active":"2024-03-19T02:54:56.192217+0000","last_became_peered":"2024-03-19T02:54:56.192217+0000","last_unstale":"2024-03-19T03:06:23.856248+0000","last_undegraded":"2024-03-19T03:06:23.856248+0000","last_fullsized":"2024-03-19T03:06:23.856248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'479","last_scrub_stamp":"2024-03-19T03:06:23.856149+0000","last_deep_scrub":"38'479","last_deep_scrub_stamp":"2024-03-19T03:06:23.856149+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.856149+0000","objects_scrubbed":161,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:49.297552+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0.016349961999999999,"stat_sum":{"num_bytes":23588150,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":30,"num_read_kb":20,"num_write":344,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"38'265","reported_seq":331,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.311495+0000","last_change":"2024-03-19T03:06:21.311546+0000","last_active":"2024-03-19T03:06:21.311495+0000","last_peered":"2024-03-19T03:06:21.311495+0000","last_clean":"2024-03-19T03:06:21.311495+0000","last_became_active":"2024-03-19T02:54:55.200182+0000","last_became_peered":"2024-03-19T02:54:55.200182+0000","last_unstale":"2024-03-19T03:06:21.311495+0000","last_undegraded":"2024-03-19T03:06:21.311495+0000","last_fullsized":"2024-03-19T03:06:21.311495+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'265","last_scrub_stamp":"2024-03-19T03:06:21.311393+0000","last_deep_scrub":"38'265","last_deep_scrub_stamp":"2024-03-19T03:06:21.311393+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.311393+0000","objects_scrubbed":3,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:26:36.352676+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":264,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23133,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"38'503","reported_seq":2629,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:28.087899+0000","last_change":"2024-03-19T03:06:28.087939+0000","last_active":"2024-03-19T03:06:28.087899+0000","last_peered":"2024-03-19T03:06:28.087899+0000","last_clean":"2024-03-19T03:06:28.087899+0000","last_became_active":"2024-03-19T02:54:56.200247+0000","last_became_peered":"2024-03-19T02:54:56.200247+0000","last_unstale":"2024-03-19T03:06:28.087899+0000","last_undegraded":"2024-03-19T03:06:28.087899+0000","last_fullsized":"2024-03-19T03:06:28.087899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'503","last_scrub_stamp":"2024-03-19T03:06:28.087800+0000","last_deep_scrub":"38'503","last_deep_scrub_stamp":"2024-03-19T03:06:28.087800+0000","last_clean_scrub_stamp":"2024-03-19T03:06:28.087800+0000","objects_scrubbed":178,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:12:41.162814+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0.061753209000000003,"stat_sum":{"num_bytes":19421231,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":34,"num_read_kb":12304,"num_write":344,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"38'80","reported_seq":148,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:23.322039+0000","last_change":"2024-03-19T03:06:23.322089+0000","last_active":"2024-03-19T03:06:23.322039+0000","last_peered":"2024-03-19T03:06:23.322039+0000","last_clean":"2024-03-19T03:06:23.322039+0000","last_became_active":"2024-03-19T02:54:55.205812+0000","last_became_peered":"2024-03-19T02:54:55.205812+0000","last_unstale":"2024-03-19T03:06:23.322039+0000","last_undegraded":"2024-03-19T03:06:23.322039+0000","last_fullsized":"2024-03-19T03:06:23.322039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'80","last_scrub_stamp":"2024-03-19T03:06:23.321939+0000","last_deep_scrub":"38'80","last_deep_scrub_stamp":"2024-03-19T03:06:23.321939+0000","last_clean_scrub_stamp":"2024-03-19T03:06:23.321939+0000","objects_scrubbed":4,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:18:36.981671+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":76,"num_write_kb":12383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"38'485","reported_seq":2599,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:31.057129+0000","last_change":"2024-03-19T03:06:31.057183+0000","last_active":"2024-03-19T03:06:31.057129+0000","last_peered":"2024-03-19T03:06:31.057129+0000","last_clean":"2024-03-19T03:06:31.057129+0000","last_became_active":"2024-03-19T02:54:56.215906+0000","last_became_peered":"2024-03-19T02:54:56.215906+0000","last_unstale":"2024-03-19T03:06:31.057129+0000","last_undegraded":"2024-03-19T03:06:31.057129+0000","last_fullsized":"2024-03-19T03:06:31.057129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'485","last_scrub_stamp":"2024-03-19T03:06:31.057044+0000","last_deep_scrub":"38'485","last_deep_scrub_stamp":"2024-03-19T03:06:31.057044+0000","last_clean_scrub_stamp":"2024-03-19T03:06:31.057044+0000","objects_scrubbed":170,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:44.230916+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0.061740428,"stat_sum":{"num_bytes":14078096,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":23,"num_read_kb":8205,"num_write":330,"num_write_kb":13929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"38'288","reported_seq":367,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.849450+0000","last_change":"2024-03-19T03:06:25.849481+0000","last_active":"2024-03-19T03:06:25.849450+0000","last_peered":"2024-03-19T03:06:25.849450+0000","last_clean":"2024-03-19T03:06:25.849450+0000","last_became_active":"2024-03-19T02:54:55.180919+0000","last_became_peered":"2024-03-19T02:54:55.180919+0000","last_unstale":"2024-03-19T03:06:25.849450+0000","last_undegraded":"2024-03-19T03:06:25.849450+0000","last_fullsized":"2024-03-19T03:06:25.849450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'288","last_scrub_stamp":"2024-03-19T03:06:25.849385+0000","last_deep_scrub":"38'288","last_deep_scrub_stamp":"2024-03-19T03:06:25.849385+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.849385+0000","objects_scrubbed":6,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:35.857810+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":40970,"num_write":274,"num_write_kb":12429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20366,"num_omap_keys":34,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"38'500","reported_seq":2618,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.706019+0000","last_change":"2024-03-19T03:06:27.706045+0000","last_active":"2024-03-19T03:06:27.706019+0000","last_peered":"2024-03-19T03:06:27.706019+0000","last_clean":"2024-03-19T03:06:27.706019+0000","last_became_active":"2024-03-19T02:54:56.192905+0000","last_became_peered":"2024-03-19T02:54:56.192905+0000","last_unstale":"2024-03-19T03:06:27.706019+0000","last_undegraded":"2024-03-19T03:06:27.706019+0000","last_fullsized":"2024-03-19T03:06:27.706019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'500","last_scrub_stamp":"2024-03-19T03:06:27.705957+0000","last_deep_scrub":"38'500","last_deep_scrub_stamp":"2024-03-19T03:06:27.705957+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.705957+0000","objects_scrubbed":176,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:40:43.653797+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0.035412037,"stat_sum":{"num_bytes":11182535,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":26,"num_read_kb":4109,"num_write":343,"num_write_kb":11053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"38'441","reported_seq":2558,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.723866+0000","last_change":"2024-03-19T03:06:27.723899+0000","last_active":"2024-03-19T03:06:27.723866+0000","last_peered":"2024-03-19T03:06:27.723866+0000","last_clean":"2024-03-19T03:06:27.723866+0000","last_became_active":"2024-03-19T02:54:56.184429+0000","last_became_peered":"2024-03-19T02:54:56.184429+0000","last_unstale":"2024-03-19T03:06:27.723866+0000","last_undegraded":"2024-03-19T03:06:27.723866+0000","last_fullsized":"2024-03-19T03:06:27.723866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'441","last_scrub_stamp":"2024-03-19T03:06:27.723805+0000","last_deep_scrub":"38'441","last_deep_scrub_stamp":"2024-03-19T03:06:27.723805+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.723805+0000","objects_scrubbed":152,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.523654+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0.053825845999999997,"stat_sum":{"num_bytes":7076024,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":303,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"38'94","reported_seq":167,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.804174+0000","last_change":"2024-03-19T03:06:26.804230+0000","last_active":"2024-03-19T03:06:26.804174+0000","last_peered":"2024-03-19T03:06:26.804174+0000","last_clean":"2024-03-19T03:06:26.804174+0000","last_became_active":"2024-03-19T02:54:55.181238+0000","last_became_peered":"2024-03-19T02:54:55.181238+0000","last_unstale":"2024-03-19T03:06:26.804174+0000","last_undegraded":"2024-03-19T03:06:26.804174+0000","last_fullsized":"2024-03-19T03:06:26.804174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'94","last_scrub_stamp":"2024-03-19T03:06:26.804070+0000","last_deep_scrub":"38'94","last_deep_scrub_stamp":"2024-03-19T03:06:26.804070+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.804070+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:21:27.125592+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":16402,"num_write":92,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":36654,"num_omap_keys":62,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"38'501","reported_seq":2616,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:26.809919+0000","last_change":"2024-03-19T03:06:26.809970+0000","last_active":"2024-03-19T03:06:26.809919+0000","last_peered":"2024-03-19T03:06:26.809919+0000","last_clean":"2024-03-19T03:06:26.809919+0000","last_became_active":"2024-03-19T02:54:56.182696+0000","last_became_peered":"2024-03-19T02:54:56.182696+0000","last_unstale":"2024-03-19T03:06:26.809919+0000","last_undegraded":"2024-03-19T03:06:26.809919+0000","last_fullsized":"2024-03-19T03:06:26.809919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'501","last_scrub_stamp":"2024-03-19T03:06:26.809802+0000","last_deep_scrub":"38'501","last_deep_scrub_stamp":"2024-03-19T03:06:26.809802+0000","last_clean_scrub_stamp":"2024-03-19T03:06:26.809802+0000","objects_scrubbed":175,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:38.011608+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0.053805551,"stat_sum":{"num_bytes":27334536,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":24,"num_read_kb":7741,"num_write":341,"num_write_kb":30967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"38'2323","reported_seq":2404,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:27.845552+0000","last_change":"2024-03-19T03:06:27.845611+0000","last_active":"2024-03-19T03:06:27.845552+0000","last_peered":"2024-03-19T03:06:27.845552+0000","last_clean":"2024-03-19T03:06:27.845552+0000","last_became_active":"2024-03-19T02:54:55.182320+0000","last_became_peered":"2024-03-19T02:54:55.182320+0000","last_unstale":"2024-03-19T03:06:27.845552+0000","last_undegraded":"2024-03-19T03:06:27.845552+0000","last_fullsized":"2024-03-19T03:06:27.845552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'2323","last_scrub_stamp":"2024-03-19T03:06:27.845449+0000","last_deep_scrub":"38'2323","last_deep_scrub_stamp":"2024-03-19T03:06:27.845449+0000","last_clean_scrub_stamp":"2024-03-19T03:06:27.845449+0000","objects_scrubbed":8,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:32.979478+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":49170,"num_write":2306,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34432,"num_omap_keys":58,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"38'442","reported_seq":2564,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:21.344844+0000","last_change":"2024-03-19T03:06:21.344877+0000","last_active":"2024-03-19T03:06:21.344844+0000","last_peered":"2024-03-19T03:06:21.344844+0000","last_clean":"2024-03-19T03:06:21.344844+0000","last_became_active":"2024-03-19T02:54:56.205954+0000","last_became_peered":"2024-03-19T02:54:56.205954+0000","last_unstale":"2024-03-19T03:06:21.344844+0000","last_undegraded":"2024-03-19T03:06:21.344844+0000","last_fullsized":"2024-03-19T03:06:21.344844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'442","last_scrub_stamp":"2024-03-19T03:06:21.344774+0000","last_deep_scrub":"38'442","last_deep_scrub_stamp":"2024-03-19T03:06:21.344774+0000","last_clean_scrub_stamp":"2024-03-19T03:06:21.344774+0000","objects_scrubbed":157,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:21.308950+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0.067674389000000001,"stat_sum":{"num_bytes":14508603,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":31,"num_read_kb":8211,"num_write":323,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"38'1555","reported_seq":1631,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:20.748237+0000","last_change":"2024-03-19T03:06:20.748341+0000","last_active":"2024-03-19T03:06:20.748237+0000","last_peered":"2024-03-19T03:06:20.748237+0000","last_clean":"2024-03-19T03:06:20.748237+0000","last_became_active":"2024-03-19T02:54:55.201077+0000","last_became_peered":"2024-03-19T02:54:55.201077+0000","last_unstale":"2024-03-19T03:06:20.748237+0000","last_undegraded":"2024-03-19T03:06:20.748237+0000","last_fullsized":"2024-03-19T03:06:20.748237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'1555","last_scrub_stamp":"2024-03-19T03:06:20.748068+0000","last_deep_scrub":"38'1555","last_deep_scrub_stamp":"2024-03-19T03:06:20.748068+0000","last_clean_scrub_stamp":"2024-03-19T03:06:20.748068+0000","objects_scrubbed":7,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:01:50.705415+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":12,"num_read_kb":42561,"num_write":1549,"num_write_kb":13574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42658,"num_omap_keys":72,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"38'458","reported_seq":2570,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:25.006073+0000","last_change":"2024-03-19T03:06:25.006146+0000","last_active":"2024-03-19T03:06:25.006073+0000","last_peered":"2024-03-19T03:06:25.006073+0000","last_clean":"2024-03-19T03:06:25.006073+0000","last_became_active":"2024-03-19T02:54:56.192515+0000","last_became_peered":"2024-03-19T02:54:56.192515+0000","last_unstale":"2024-03-19T03:06:25.006073+0000","last_undegraded":"2024-03-19T03:06:25.006073+0000","last_fullsized":"2024-03-19T03:06:25.006073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'458","last_scrub_stamp":"2024-03-19T03:06:25.005899+0000","last_deep_scrub":"38'458","last_deep_scrub_stamp":"2024-03-19T03:06:25.005899+0000","last_clean_scrub_stamp":"2024-03-19T03:06:25.005899+0000","objects_scrubbed":158,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:48:42.528973+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0.065375344000000002,"stat_sum":{"num_bytes":15008616,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":21,"num_read_kb":8204,"num_write":333,"num_write_kb":14782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"38'154","reported_seq":240,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-19T03:06:18.199549+0000","last_change":"2024-03-19T03:06:18.199587+0000","last_active":"2024-03-19T03:06:18.199549+0000","last_peered":"2024-03-19T03:06:18.199549+0000","last_clean":"2024-03-19T03:06:18.199549+0000","last_became_active":"2024-03-19T02:54:55.199414+0000","last_became_peered":"2024-03-19T02:54:55.199414+0000","last_unstale":"2024-03-19T03:06:18.199549+0000","last_undegraded":"2024-03-19T03:06:18.199549+0000","last_fullsized":"2024-03-19T03:06:18.199549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'154","last_scrub_stamp":"2024-03-19T03:06:18.199476+0000","last_deep_scrub":"38'154","last_deep_scrub_stamp":"2024-03-19T03:06:18.199476+0000","last_clean_scrub_stamp":"2024-03-19T03:06:18.199476+0000","objects_scrubbed":6,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.265804+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":45076,"num_write":147,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20366,"num_omap_keys":34,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":939466941,"num_objects":10564,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10564,"num_whiteouts":0,"num_read":1809,"num_read_kb":363000,"num_write":21148,"num_write_kb":1062314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1709039616,"data_stored":1877393786,"data_compressed":164809216,"data_compressed_allocated":182280192,"data_compressed_original":405618688,"omap_allocated":0,"internal_metadata":0},"log_size":30368,"ondisk_log_size":30368,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":233417001,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":557,"num_read_kb":727830,"num_write":11958,"num_write_kb":238649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":582147,"num_omap_keys":982,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123199488,"data_stored":466834002,"data_compressed":55225082,"data_compressed_allocated":70197248,"data_compressed_original":413892608,"omap_allocated":0,"internal_metadata":0},"log_size":11770,"ondisk_log_size":11770,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64792,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574992,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336340,"kb_used_data":242700,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94035500,"statfs":{"total":96636764160,"available":96292352000,"internally_reserved":0,"allocated":248524800,"data_stored":332229150,"data_compressed":32896566,"data_compressed_allocated":38154240,"data_compressed_original":130297856,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574993,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288080,"kb_used_data":194504,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94083760,"statfs":{"total":96636764160,"available":96341770240,"internally_reserved":0,"allocated":199172096,"data_stored":260537624,"data_compressed":19587928,"data_compressed_allocated":23076864,"data_compressed_original":89665536,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353596,"kb_used_data":254388,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94018244,"statfs":{"total":96636764160,"available":96274681856,"internally_reserved":0,"allocated":260493312,"data_stored":321864354,"data_compressed":24348322,"data_compressed_allocated":28299264,"data_compressed_original":97570816,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329372,"kb_used_data":253780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94042468,"statfs":{"total":96636764160,"available":96299487232,"internally_reserved":0,"allocated":259870720,"data_stored":336209440,"data_compressed":37546180,"data_compressed_allocated":42463232,"data_compressed_original":127148032,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347780,"kb_used_data":276348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94024060,"statfs":{"total":96636764160,"available":96280637440,"internally_reserved":0,"allocated":282980352,"data_stored":343276606,"data_compressed":27693759,"data_compressed_allocated":31715328,"data_compressed_original":100061184,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188740,"kb_used_data":139516,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94183100,"statfs":{"total":96636764160,"available":96443494400,"internally_reserved":0,"allocated":142864384,"data_stored":174161585,"data_compressed":16362216,"data_compressed_allocated":18694144,"data_compressed_original":54902784,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":217160,"kb_used_data":167936,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94154680,"statfs":{"total":96636764160,"available":96414392320,"internally_reserved":0,"allocated":171966464,"data_stored":244957439,"data_compressed":28916562,"data_compressed_allocated":33230848,"data_compressed_original":111853568,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335012,"kb_used_data":263580,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94036828,"statfs":{"total":96636764160,"available":96293711872,"internally_reserved":0,"allocated":269905920,"data_stored":333709622,"data_compressed":32747557,"data_compressed_allocated":36933632,"data_compressed_original":109158400,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32396,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12947456,"data_stored":50332286,"data_compressed":6253471,"data_compressed_allocated":7770112,"data_compressed_original":45166592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14929920,"data_stored":65109342,"data_compressed":8347451,"data_compressed_allocated":10539008,"data_compressed_original":60727296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":21505709,"data_compressed":2290904,"data_compressed_allocated":3022848,"data_compressed_original":17223680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11972608,"data_stored":50866509,"data_compressed":6550764,"data_compressed_allocated":8245248,"data_compressed_original":47149056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14008320,"data_stored":62914686,"data_compressed":7417000,"data_compressed_allocated":9461760,"data_compressed_original":58380288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21315584,"data_stored":65109272,"data_compressed":7319535,"data_compressed_allocated":9244672,"data_compressed_original":53043200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20590592,"data_stored":71303840,"data_compressed":7958306,"data_compressed_allocated":10170368,"data_compressed_original":60891136,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":79692358,"data_compressed":9087651,"data_compressed_allocated":11743232,"data_compressed_original":71311360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256466944,"data_stored":282594806,"data_compressed":26461690,"data_compressed_allocated":29118464,"data_compressed_original":63418368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156610560,"data_stored":179655935,"data_compressed":20569111,"data_compressed_allocated":22691840,"data_compressed_original":51126272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135131136,"data_stored":152463714,"data_compressed":14071312,"data_compressed_allocated":15671296,"data_compressed_original":37679104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270581760,"data_stored":292217935,"data_compressed":21142995,"data_compressed_allocated":23470080,"data_compressed_original":52912128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245436416,"data_stored":273102592,"data_compressed":30129180,"data_compressed_allocated":33001472,"data_compressed_original":68767744,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238751744,"data_stored":256562920,"data_compressed":17028787,"data_compressed_allocated":19054592,"data_compressed_original":44527616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178155520,"data_stored":189041622,"data_compressed":11629622,"data_compressed_allocated":12906496,"data_compressed_original":28774400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":227905536,"data_stored":251754262,"data_compressed":23776519,"data_compressed_allocated":26365952,"data_compressed_original":58413056,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:06:42.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-19T03:06:43.145 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.144+0000 7f6e96800640 1 Processor -- start 2024-03-19T03:06:43.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.144+0000 7f6e96800640 1 -- start start 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.145+0000 7f6e96800640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.145+0000 7f6e96800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.145+0000 7f6e96800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e900caf90 con 0x7f6e90067ad0 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.145+0000 7f6e96800640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f6e9015e280 con 0x7f6e90154cd0 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.145+0000 7f6e96800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9015e520 con 0x7f6e901550b0 2024-03-19T03:06:43.148 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52444/0 (socket says 172.21.15.92:52444) 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8f400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40534/0 (socket says 172.21.15.92:40534) 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 -- 172.21.15.92:0/1880512091 learned_addr learned my addr 172.21.15.92:0/1880512091 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8fe00640 1 --1- 172.21.15.92:0/1880512091 >> v1:172.21.15.92:6789/0 conn(0x7f6e90067ad0 0x7f6e900cbfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:44514/0 (socket says 172.21.15.92:44514) 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 -- 172.21.15.92:0/1880512091 >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 msgr2=0x7f6e9015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 --2- 172.21.15.92:0/1880512091 >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 -- 172.21.15.92:0/1880512091 >> v1:172.21.15.92:6789/0 conn(0x7f6e90067ad0 legacy=0x7f6e900cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8ea00640 1 -- 172.21.15.92:0/1880512091 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9015e800 con 0x7f6e901550b0 2024-03-19T03:06:43.150 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.146+0000 7f6e8f400640 1 --2- 172.21.15.92:0/1880512091 >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.147+0000 7f6e8ea00640 1 --2- 172.21.15.92:0/1880512091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009870 tx=0x7f6e7c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f1edcd12a1faa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.147+0000 7f6e8e000640 1 -- 172.21.15.92:0/1880512091 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c034070 con 0x7f6e901550b0 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e8e000640 1 -- 172.21.15.92:0/1880512091 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c030840 con 0x7f6e901550b0 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e8e000640 1 -- 172.21.15.92:0/1880512091 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c030c10 con 0x7f6e901550b0 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 -- 172.21.15.92:0/1880512091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 msgr2=0x7f6e900cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.151 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 --2- 172.21.15.92:0/1880512091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009870 tx=0x7f6e7c02b980 comp rx=0 tx=0).stop 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 -- 172.21.15.92:0/1880512091 shutdown_connections 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 --2- 172.21.15.92:0/1880512091 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e901550b0 0x7f6e900cceb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 --2- 172.21.15.92:0/1880512091 >> v2:172.21.15.159:3300/0 conn(0x7f6e90154cd0 0x7f6e9015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 -- 172.21.15.92:0/1880512091 >> 172.21.15.92:0/1880512091 conn(0x7f6e90072970 msgr2=0x7f6e90066f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 -- 172.21.15.92:0/1880512091 shutdown_connections 2024-03-19T03:06:43.152 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.148+0000 7f6e96800640 1 -- 172.21.15.92:0/1880512091 wait complete. 2024-03-19T03:06:43.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.150+0000 7f6e96800640 1 Processor -- start 2024-03-19T03:06:43.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.150+0000 7f6e96800640 1 -- start start 2024-03-19T03:06:43.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e901550b0 0x7f6e9014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6e900cc6b0 con 0x7f6e901550b0 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9015edc0 con 0x7f6e90067ad0 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e96800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6e900cc830 con 0x7f6e90154cd0 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8f400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40542/0 (socket says 172.21.15.92:40542) 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8ea00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52452/0 (socket says 172.21.15.92:52452) 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8f400640 1 -- 172.21.15.92:0/166710898 learned_addr learned my addr 172.21.15.92:0/166710898 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.151+0000 7f6e8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e901550b0 0x7f6e9014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 -- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 msgr2=0x7f6e901574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 -- 172.21.15.92:0/166710898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e901550b0 msgr2=0x7f6e9014fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e901550b0 0x7f6e9014fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 -- 172.21.15.92:0/166710898 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9014f050 con 0x7f6e90067ad0 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8ea00640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e8f400640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000e990 tx=0x7f6e8000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80017070 con 0x7f6e90067ad0 2024-03-19T03:06:43.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8000c9e0 con 0x7f6e90067ad0 2024-03-19T03:06:43.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8000cdb0 con 0x7f6e90067ad0 2024-03-19T03:06:43.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.152+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e90157cd0 con 0x7f6e90067ad0 2024-03-19T03:06:43.157 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.153+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e9015a860 con 0x7f6e90067ad0 2024-03-19T03:06:43.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.154+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e900c71f0 con 0x7f6e90067ad0 2024-03-19T03:06:43.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.154+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6e8001a070 con 0x7f6e90067ad0 2024-03-19T03:06:43.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.155+0000 7f6e87400640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.158 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.155+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f6e8008ae90 con 0x7f6e90067ad0 2024-03-19T03:06:43.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.156+0000 7f6e8ea00640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.162 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.156+0000 7f6e8ea00640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 0x7f6e7006e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c036000 tx=0x7f6e7c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.163 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.160+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6e900c71f0 con 0x7f6e90067ad0 2024-03-19T03:06:43.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.359+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f6e900ca8f0 con 0x7f6e90067ad0 2024-03-19T03:06:43.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.367+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6e800590e0 con 0x7f6e90067ad0 2024-03-19T03:06:43.369 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.368+0000 7f6e87400640 1 -- 172.21.15.92:0/166710898 <== mon.1 v2:172.21.15.159:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f6e8005d180 con 0x7f6e90067ad0 2024-03-19T03:06:43.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 msgr2=0x7f6e7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 0x7f6e7006e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c036000 tx=0x7f6e7c03b040 comp rx=0 tx=0).stop 2024-03-19T03:06:43.373 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 msgr2=0x7f6e90156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000e990 tx=0x7f6e8000ee50 comp rx=0 tx=0).stop 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 shutdown_connections 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e901550b0 0x7f6e9014fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e90154cd0 0x7f6e901574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 --2- 172.21.15.92:0/166710898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6e90067ad0 0x7f6e90156f70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.374 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 >> 172.21.15.92:0/166710898 conn(0x7f6e90072970 msgr2=0x7f6e9015bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 shutdown_connections 2024-03-19T03:06:43.375 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.371+0000 7f6e96800640 1 -- 172.21.15.92:0/166710898 wait complete. 2024-03-19T03:06:43.392 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T03:06:43.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:06:43.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.590+0000 7fac06200640 1 Processor -- start 2024-03-19T03:06:43.592 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.590+0000 7fac06200640 1 -- start start 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac00152bf0 0x7fac0006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 0x7fac000c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac000c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fac000c6c90 con 0x7fac000c6470 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fac00159910 con 0x7fac00152bf0 2024-03-19T03:06:43.594 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.591+0000 7fac06200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fac0006e4a0 con 0x7fac001593d0 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac00152bf0 0x7fac0006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac00152bf0 0x7fac0006eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40550/0 (socket says 172.21.15.92:40550) 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 -- 172.21.15.92:0/1132750588 learned_addr learned my addr 172.21.15.92:0/1132750588 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabffe00640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac000c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabf7e00640 1 --2- 172.21.15.92:0/1132750588 >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 0x7fac000c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 -- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 msgr2=0x7fac000c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac000c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 -- 172.21.15.92:0/1132750588 >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 msgr2=0x7fac000c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:43.595 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 --2- 172.21.15.92:0/1132750588 >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 0x7fac000c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabff400640 1 -- 172.21.15.92:0/1132750588 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac000c9a80 con 0x7fac00152bf0 2024-03-19T03:06:43.596 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.592+0000 7fabf7e00640 1 --2- 172.21.15.92:0/1132750588 >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 0x7fac000c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.593+0000 7fabff400640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac00152bf0 0x7fac0006eef0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fac00075180 tx=0x7fabf002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0d983a1ad742e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fabfea00640 1 -- 172.21.15.92:0/1132750588 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf002e070 con 0x7fac00152bf0 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fabfea00640 1 -- 172.21.15.92:0/1132750588 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fabf0037070 con 0x7fac00152bf0 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fabfea00640 1 -- 172.21.15.92:0/1132750588 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0031a20 con 0x7fac00152bf0 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 -- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac00152bf0 msgr2=0x7fac0006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.597 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac00152bf0 0x7fac0006eef0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fac00075180 tx=0x7fabf002b880 comp rx=0 tx=0).stop 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 -- 172.21.15.92:0/1132750588 shutdown_connections 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac000c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 --2- 172.21.15.92:0/1132750588 >> v2:172.21.15.92:3300/0 conn(0x7fac000c6470 0x7fac000c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 --2- 172.21.15.92:0/1132750588 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac00152bf0 0x7fac0006eef0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 -- 172.21.15.92:0/1132750588 >> 172.21.15.92:0/1132750588 conn(0x7fac000c8d90 msgr2=0x7fac000c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 -- 172.21.15.92:0/1132750588 shutdown_connections 2024-03-19T03:06:43.598 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.594+0000 7fac06200640 1 -- 172.21.15.92:0/1132750588 wait complete. 2024-03-19T03:06:43.600 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.596+0000 7fac06200640 1 Processor -- start 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.596+0000 7fac06200640 1 -- start start 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac000c6470 0x7fac0015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac00152bf0 0x7fac001f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fac000c8930 con 0x7fac00152bf0 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fac0006ea60 con 0x7fac000c6470 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fac06200640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fac0006e6f0 con 0x7fac001593d0 2024-03-19T03:06:43.601 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac00152bf0 0x7fac001f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fabffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fabff400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac000c6470 0x7fac0015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fabffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52460/0 (socket says 172.21.15.92:52460) 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.597+0000 7fabffe00640 1 -- 172.21.15.92:0/2964033833 learned_addr learned my addr 172.21.15.92:0/2964033833 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 -- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac000c6470 msgr2=0x7fac0015b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac000c6470 0x7fac0015b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 -- 172.21.15.92:0/2964033833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac00152bf0 msgr2=0x7fac001f1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac00152bf0 0x7fac001f1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 -- 172.21.15.92:0/2964033833 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac001f4680 con 0x7fac001593d0 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabffe00640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009cb0 tx=0x7fabf800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8018020 con 0x7fac001593d0 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fabf8014480 con 0x7fac001593d0 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8014890 con 0x7fac001593d0 2024-03-19T03:06:43.603 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac001f0bd0 con 0x7fac001593d0 2024-03-19T03:06:43.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.598+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac001f4bf0 con 0x7fac001593d0 2024-03-19T03:06:43.604 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.599+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac000684c0 con 0x7fac001593d0 2024-03-19T03:06:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.600+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fabf8005d10 con 0x7fac001593d0 2024-03-19T03:06:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.601+0000 7fabf7400640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 0x7fabdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.601+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7fabf808b9e0 con 0x7fac001593d0 2024-03-19T03:06:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.601+0000 7fabff400640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 0x7fabdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.605 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.602+0000 7fabff400640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 0x7fabdc06e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fabf0036000 tx=0x7fabf003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.605+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fabf8022030 con 0x7fac001593d0 2024-03-19T03:06:43.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.843+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac000c3b50 con 0x7fac001593d0 2024-03-19T03:06:43.846 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.844+0000 7fabf7400640 1 -- 172.21.15.92:0/2964033833 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6839 (secure 0 0 0) 0x7fabf8010040 con 0x7fac001593d0 2024-03-19T03:06:43.847 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:43.847 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5506,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3348696376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3348696376},{"type":"v1","addr":"172.21.15.159:6839","nonce":3348696376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:41.531709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5347,"qdb_cluster":[5347]},"id":1}]} 2024-03-19T03:06:43.847 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 37 2024-03-19T03:06:43.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.847+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 msgr2=0x7fabdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.847+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 0x7fabdc06e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fabf0036000 tx=0x7fabf003d040 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.847+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 msgr2=0x7fac001f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.847+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009cb0 tx=0x7fabf800c490 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 shutdown_connections 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fabdc06bbd0 0x7fabdc06e090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac001593d0 0x7fac001f4140 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac00152bf0 0x7fac001f1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 --2- 172.21.15.92:0/2964033833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac000c6470 0x7fac0015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 >> 172.21.15.92:0/2964033833 conn(0x7fac000c8d90 msgr2=0x7fac000c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 shutdown_connections 2024-03-19T03:06:43.851 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:43.848+0000 7fac06200640 1 -- 172.21.15.92:0/2964033833 wait complete. 2024-03-19T03:06:43.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:06:44.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.069+0000 7f7c31a00640 1 Processor -- start 2024-03-19T03:06:44.070 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.069+0000 7f7c31a00640 1 -- start start 2024-03-19T03:06:44.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c2c15ea70 0x7f7c2c0c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31a00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 0x7f7c2c0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2c15ef60 con 0x7f7c2c154e00 2024-03-19T03:06:44.072 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7c2c0c6bf0 con 0x7f7c2c15ea70 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31a00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7c2c072610 con 0x7f7c2c15b550 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c2be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7c2c154e00 0x7f7c2c15ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:44528/0 (socket says 172.21.15.92:44528) 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c2be00640 1 -- 172.21.15.92:0/3369160180 learned_addr learned my addr 172.21.15.92:0/3369160180 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c23400640 1 --2- 172.21.15.92:0/3369160180 >> v2:172.21.15.92:3300/0 conn(0x7f7c2c15ea70 0x7f7c2c0c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31000640 1 --2- 172.21.15.92:0/3369160180 >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 0x7f7c2c0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c2b400640 1 -- 172.21.15.92:0/3369160180 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326135919 0 0) 0x7f7c2c15ef60 con 0x7f7c2c154e00 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c2b400640 1 -- 172.21.15.92:0/3369160180 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c10003520 con 0x7f7c2c154e00 2024-03-19T03:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c23400640 1 -- 172.21.15.92:0/3369160180 >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 msgr2=0x7f7c2c0c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c23400640 1 --2- 172.21.15.92:0/3369160180 >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 0x7f7c2c0c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c23400640 1 -- 172.21.15.92:0/3369160180 >> v1:172.21.15.92:6789/0 conn(0x7f7c2c154e00 legacy=0x7f7c2c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c23400640 1 -- 172.21.15.92:0/3369160180 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c0728b0 con 0x7f7c2c15ea70 2024-03-19T03:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.070+0000 7f7c31000640 1 --2- 172.21.15.92:0/3369160180 >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 0x7f7c2c0c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.071+0000 7f7c23400640 1 --2- 172.21.15.92:0/3369160180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c0c7490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c068220 tx=0x7f7c1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbffcd6abe1dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c2b400640 1 -- 172.21.15.92:0/3369160180 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14034070 con 0x7f7c2c15ea70 2024-03-19T03:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c2b400640 1 -- 172.21.15.92:0/3369160180 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7c14030610 con 0x7f7c2c15ea70 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c2b400640 1 -- 172.21.15.92:0/3369160180 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14030a00 con 0x7f7c2c15ea70 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c31a00640 1 -- 172.21.15.92:0/3369160180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 msgr2=0x7f7c2c0c7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c31a00640 1 --2- 172.21.15.92:0/3369160180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c0c7490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c068220 tx=0x7f7c1402b960 comp rx=0 tx=0).stop 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.072+0000 7f7c31a00640 1 -- 172.21.15.92:0/3369160180 shutdown_connections 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.073+0000 7f7c31a00640 1 --2- 172.21.15.92:0/3369160180 >> v2:172.21.15.159:3300/0 conn(0x7f7c2c15b550 0x7f7c2c0c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.073+0000 7f7c31a00640 1 --2- 172.21.15.92:0/3369160180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c0c7490 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.073+0000 7f7c31a00640 1 -- 172.21.15.92:0/3369160180 >> 172.21.15.92:0/3369160180 conn(0x7f7c2c0cc1f0 msgr2=0x7f7c2c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.073+0000 7f7c31a00640 1 -- 172.21.15.92:0/3369160180 shutdown_connections 2024-03-19T03:06:44.077 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.073+0000 7f7c31a00640 1 -- 172.21.15.92:0/3369160180 wait complete. 2024-03-19T03:06:44.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.074+0000 7f7c31a00640 1 Processor -- start 2024-03-19T03:06:44.079 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.074+0000 7f7c31a00640 1 -- start start 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c2c15b550 0x7f7c2c146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c15d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c1551e0 con 0x7f7c2c15ea70 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0c7120 con 0x7f7c2c154e00 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c31a00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0c7f10 con 0x7f7c2c15b550 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.075+0000 7f7c2be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.080 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c31000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c15d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40574/0 (socket says 172.21.15.92:40574) 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 -- 172.21.15.92:0/672494935 learned_addr learned my addr 172.21.15.92:0/672494935 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c23400640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c2c15b550 0x7f7c2c146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 -- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c2c15b550 msgr2=0x7f7c2c146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c2c15b550 0x7f7c2c146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 -- 172.21.15.92:0/672494935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 msgr2=0x7f7c2c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c15d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 -- 172.21.15.92:0/672494935 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c15d690 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c2be00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c390 tx=0x7f7c1c00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.076+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c017070 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.077+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7c1c005ce0 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.077+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c1474b0 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.077+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c007310 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.077+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c14c540 con 0x7f7c2c154e00 2024-03-19T03:06:44.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.078+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c04f580 con 0x7f7c2c154e00 2024-03-19T03:06:44.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.079+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7c1c01a070 con 0x7f7c2c154e00 2024-03-19T03:06:44.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.080+0000 7f7c28c00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.083 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.080+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f7c1c08a050 con 0x7f7c2c154e00 2024-03-19T03:06:44.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.080+0000 7f7c23400640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.084 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.081+0000 7f7c23400640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 0x7f7c0806e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7c14036000 tx=0x7f7c1403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.089+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7c2c1474b0 con 0x7f7c2c154e00 2024-03-19T03:06:44.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.326+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2c0c3640 con 0x7f7c2c154e00 2024-03-19T03:06:44.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.327+0000 7f7c28c00640 1 -- 172.21.15.92:0/672494935 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6839 (secure 0 0 0) 0x7f7c2c15d690 con 0x7f7c2c154e00 2024-03-19T03:06:44.329 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:44.329 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4188,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/4145321899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":4145321899},{"type":"v1","addr":"172.21.15.92:6835","nonce":4145321899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4197,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2921443021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2921443021},{"type":"v1","addr":"172.21.15.92:6837","nonce":2921443021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/3633923980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3633923980},{"type":"v1","addr":"172.21.15.159:6836","nonce":3633923980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/3249321834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3249321834},{"type":"v1","addr":"172.21.15.92:6833","nonce":3249321834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5506,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6839/3348696376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6838","nonce":3348696376},{"type":"v1","addr":"172.21.15.159:6839","nonce":3348696376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:54:55.655938+0000","modified":"2024-03-19T03:05:41.531709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5347":{"gid":5347,"name":"f","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.159:6837/3372688839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":3372688839},{"type":"v1","addr":"172.21.15.159:6837","nonce":3372688839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5347,"qdb_cluster":[5347]},"id":1}]} 2024-03-19T03:06:44.330 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 37 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 msgr2=0x7f7c0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 0x7f7c0806e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7c14036000 tx=0x7f7c1403b040 comp rx=0 tx=0).stop 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 msgr2=0x7f7c2c1f1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c390 tx=0x7f7c1c00c790 comp rx=0 tx=0).stop 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 shutdown_connections 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2c15ea70 0x7f7c2c15d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c2c15b550 0x7f7c2c146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 --2- 172.21.15.92:0/672494935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c2c154e00 0x7f7c2c1f1180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 >> 172.21.15.92:0/672494935 conn(0x7f7c2c0cc1f0 msgr2=0x7f7c2c0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 shutdown_connections 2024-03-19T03:06:44.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.330+0000 7f7c31a00640 1 -- 172.21.15.92:0/672494935 wait complete. 2024-03-19T03:06:44.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:06:44.554 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.553+0000 7f4e93600640 1 Processor -- start 2024-03-19T03:06:44.555 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.553+0000 7f4e93600640 1 -- start start 2024-03-19T03:06:44.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e93600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4e8c151900 0x7f4e8c158320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e93600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c0c3b80 con 0x7f4e8c151ce0 2024-03-19T03:06:44.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e93600640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c072e70 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e93600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8c158860 con 0x7f4e8c151900 2024-03-19T03:06:44.556 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e91600640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4e8c0c7da0 0x7f4e8c162140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.92:33448/0 (socket says 172.21.15.92:33448) 2024-03-19T03:06:44.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e90c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4e8c151900 0x7f4e8c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.557 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.554+0000 7f4e91600640 1 -- 172.21.15.92:0/2423599017 learned_addr learned my addr 172.21.15.92:0/2423599017 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879754312 0 0) 0x7f4e8c072e70 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e70003520 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703517841 0 0) 0x7f4e8c0c3b80 con 0x7f4e8c151ce0 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e90c00640 1 -- 172.21.15.92:0/2423599017 >> v1:172.21.15.159:6789/0 conn(0x7f4e8c0c7da0 legacy=0x7f4e8c162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e90c00640 1 -- 172.21.15.92:0/2423599017 >> v1:172.21.15.92:6789/0 conn(0x7f4e8c151ce0 legacy=0x7f4e8c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.558 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.555+0000 7f4e90c00640 1 -- 172.21.15.92:0/2423599017 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c073150 con 0x7f4e8c151900 2024-03-19T03:06:44.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.556+0000 7f4e90c00640 1 --2- 172.21.15.92:0/2423599017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151900 0x7f4e8c158320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f346944b704cc8c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.556+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c02f810 con 0x7f4e8c151900 2024-03-19T03:06:44.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.556+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4e7c02f9b0 con 0x7f4e8c151900 2024-03-19T03:06:44.559 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.556+0000 7f4e8be00640 1 -- 172.21.15.92:0/2423599017 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c02fda0 con 0x7f4e8c151900 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 -- 172.21.15.92:0/2423599017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151900 msgr2=0x7f4e8c158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 --2- 172.21.15.92:0/2423599017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151900 0x7f4e8c158320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b9f0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 -- 172.21.15.92:0/2423599017 shutdown_connections 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 --2- 172.21.15.92:0/2423599017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151900 0x7f4e8c158320 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 -- 172.21.15.92:0/2423599017 >> 172.21.15.92:0/2423599017 conn(0x7f4e8c0c8d90 msgr2=0x7f4e8c0c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 -- 172.21.15.92:0/2423599017 shutdown_connections 2024-03-19T03:06:44.560 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.557+0000 7f4e93600640 1 -- 172.21.15.92:0/2423599017 wait complete. 2024-03-19T03:06:44.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.559+0000 7f4e93600640 1 Processor -- start 2024-03-19T03:06:44.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.559+0000 7f4e93600640 1 -- start start 2024-03-19T03:06:44.563 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c151900 0x7f4e8c1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 0x7f4e8c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c0c9af0 con 0x7f4e8c151ce0 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c071eb0 con 0x7f4e8c151900 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e93600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c0c9c70 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e90c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e91600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 0x7f4e8c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.564 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e90c00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52488/0 (socket says 172.21.15.92:52488) 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e90c00640 1 -- 172.21.15.92:0/46169484 learned_addr learned my addr 172.21.15.92:0/46169484 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.560+0000 7f4e83e00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c151900 0x7f4e8c1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 -- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c151900 msgr2=0x7f4e8c1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c151900 0x7f4e8c1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 -- 172.21.15.92:0/46169484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 msgr2=0x7f4e8c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 0x7f4e8c145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.565 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 -- 172.21.15.92:0/46169484 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c1f4750 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e91600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 0x7f4e8c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e90c00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0098c0 tx=0x7f4e7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c047070 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4e7c0378d0 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c1f44b0 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c037cc0 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.566 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.561+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c159800 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.567 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.563+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c0684c0 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.563+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4e7c04a070 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.564+0000 7f4e89600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 0x7f4e6806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.564+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f4e7c0ac910 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.564+0000 7f4e83e00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 0x7f4e6806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.568 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.565+0000 7f4e83e00640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 0x7f4e6806e0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c1556e0 tx=0x7f4e74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.570 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.569+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4e7c033030 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.765 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.763+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e8c071600 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.766 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.765+0000 7f4e89600640 1 -- 172.21.15.92:0/46169484 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+17584 (secure 0 0 0) 0x7f4e7c07ab60 con 0x7f4e8c0c7da0 2024-03-19T03:06:44.767 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-19T03:06:44.767 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":38,"fsid":"91932042-1f9e-4210-9e82-70c11e6277fd","created":"2024-03-19T02:54:39.610512+0000","modified":"2024-03-19T03:05:09.726374+0000","last_up_change":"2024-03-19T02:54:49.854498+0000","last_in_change":"2024-03-19T02:54:42.054121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:54:49.925691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:54:53.864988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:54:54.662544+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7ad93e3-f661-456e-953f-d55b62cefef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6817","nonce":3305119790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6822","nonce":3305119790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6830","nonce":3305119790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":3305119790},{"type":"v1","addr":"172.21.15.92:6826","nonce":3305119790}]},"public_addr":"172.21.15.92:6817/3305119790","cluster_addr":"172.21.15.92:6822/3305119790","heartbeat_back_addr":"172.21.15.92:6830/3305119790","heartbeat_front_addr":"172.21.15.92:6826/3305119790","state":["exists","up"]},{"osd":1,"uuid":"e60c183e-d354-46db-8324-3b1fd36c5c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6809","nonce":3121211588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6811","nonce":3121211588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6816","nonce":3121211588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3121211588},{"type":"v1","addr":"172.21.15.92:6813","nonce":3121211588}]},"public_addr":"172.21.15.92:6809/3121211588","cluster_addr":"172.21.15.92:6811/3121211588","heartbeat_back_addr":"172.21.15.92:6816/3121211588","heartbeat_front_addr":"172.21.15.92:6813/3121211588","state":["exists","up"]},{"osd":2,"uuid":"0f53fb99-a1cb-4f41-a48e-22949b3fadcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6801","nonce":652641961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6803","nonce":652641961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6807","nonce":652641961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":652641961},{"type":"v1","addr":"172.21.15.92:6805","nonce":652641961}]},"public_addr":"172.21.15.92:6801/652641961","cluster_addr":"172.21.15.92:6803/652641961","heartbeat_back_addr":"172.21.15.92:6807/652641961","heartbeat_front_addr":"172.21.15.92:6805/652641961","state":["exists","up"]},{"osd":3,"uuid":"b129c2e0-714e-48bc-96d4-307931384f8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6819","nonce":1598837651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6823","nonce":1598837651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6831","nonce":1598837651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1598837651},{"type":"v1","addr":"172.21.15.92:6827","nonce":1598837651}]},"public_addr":"172.21.15.92:6819/1598837651","cluster_addr":"172.21.15.92:6823/1598837651","heartbeat_back_addr":"172.21.15.92:6831/1598837651","heartbeat_front_addr":"172.21.15.92:6827/1598837651","state":["exists","up"]},{"osd":4,"uuid":"d45a9953-14a3-47a4-9f53-6c7f7bac8b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6809","nonce":4013217604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6811","nonce":4013217604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6815","nonce":4013217604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":4013217604},{"type":"v1","addr":"172.21.15.159:6813","nonce":4013217604}]},"public_addr":"172.21.15.159:6809/4013217604","cluster_addr":"172.21.15.159:6811/4013217604","heartbeat_back_addr":"172.21.15.159:6815/4013217604","heartbeat_front_addr":"172.21.15.159:6813/4013217604","state":["exists","up"]},{"osd":5,"uuid":"aba312ec-c3d6-4903-9b99-37edcedc8ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6825","nonce":1484023345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6827","nonce":1484023345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6831","nonce":1484023345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":1484023345},{"type":"v1","addr":"172.21.15.159:6829","nonce":1484023345}]},"public_addr":"172.21.15.159:6825/1484023345","cluster_addr":"172.21.15.159:6827/1484023345","heartbeat_back_addr":"172.21.15.159:6831/1484023345","heartbeat_front_addr":"172.21.15.159:6829/1484023345","state":["exists","up"]},{"osd":6,"uuid":"b3feeff9-5f92-4ca5-8d19-c1c46e993fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6801","nonce":2808929873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6803","nonce":2808929873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6807","nonce":2808929873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":2808929873},{"type":"v1","addr":"172.21.15.159:6805","nonce":2808929873}]},"public_addr":"172.21.15.159:6801/2808929873","cluster_addr":"172.21.15.159:6803/2808929873","heartbeat_back_addr":"172.21.15.159:6807/2808929873","heartbeat_front_addr":"172.21.15.159:6805/2808929873","state":["exists","up"]},{"osd":7,"uuid":"676a4526-ecfc-4948-bfcd-d0b022bcfa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6817","nonce":3107756136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6819","nonce":3107756136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6823","nonce":3107756136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":3107756136},{"type":"v1","addr":"172.21.15.159:6821","nonce":3107756136}]},"public_addr":"172.21.15.159:6817/3107756136","cluster_addr":"172.21.15.159:6819/3107756136","heartbeat_back_addr":"172.21.15.159:6823/3107756136","heartbeat_front_addr":"172.21.15.159:6821/3107756136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.480545+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.329376+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.049214+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.354489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.204099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.307169+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.132324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:54:47.283327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6839/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6838/3845908330":"2024-03-20T03:05:09.726289+0000","172.21.15.159:6837/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6838/2444776025":"2024-03-20T03:00:59.655927+0000","172.21.15.159:6837/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6835/2066707194":"2024-03-20T03:03:24.698071+0000","172.21.15.159:6839/1521694689":"2024-03-20T02:57:04.595375+0000","172.21.15.159:6835/2387435110":"2024-03-20T02:59:24.630258+0000","172.21.15.159:6838/1521694689":"2024-03-20T02:57:04.595375+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 msgr2=0x7f4e6806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 0x7f4e6806e0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c1556e0 tx=0x7f4e74009040 comp rx=0 tx=0).stop 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 msgr2=0x7f4e8c15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0098c0 tx=0x7f4e7c004730 comp rx=0 tx=0).stop 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 shutdown_connections 2024-03-19T03:06:44.770 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f4e6806bc00 0x7f4e6806e0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e8c151ce0 0x7f4e8c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e8c151900 0x7f4e8c1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 --2- 172.21.15.92:0/46169484 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e8c0c7da0 0x7f4e8c15a000 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 >> 172.21.15.92:0/46169484 conn(0x7f4e8c0c8d90 msgr2=0x7f4e8c0c31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 shutdown_connections 2024-03-19T03:06:44.771 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.768+0000 7f4e93600640 1 -- 172.21.15.92:0/46169484 wait complete. 2024-03-19T03:06:44.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T03:06:44.988 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.988+0000 7f1d07800640 1 Processor -- start 2024-03-19T03:06:44.989 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.988+0000 7f1d07800640 1 -- start start 2024-03-19T03:06:44.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d07800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1d00071780 0x7f1d001578f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d07800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 0x7f1d00151300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d07800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d000c8a90 con 0x7f1d00071ed0 2024-03-19T03:06:44.990 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d07800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1d000c3f40 con 0x7f1d00071780 2024-03-19T03:06:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d07800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1d000c3980 con 0x7f1d000c8310 2024-03-19T03:06:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d04e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1d00071ed0 0x7f1d000722b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:44540/0 (socket says 172.21.15.92:44540) 2024-03-19T03:06:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d04e00640 1 -- 172.21.15.92:0/2822018719 learned_addr learned my addr 172.21.15.92:0/2822018719 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.989+0000 7f1d05800640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 0x7f1d00151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cffe00640 1 --2- 172.21.15.92:0/2822018719 >> v2:172.21.15.92:3300/0 conn(0x7f1d00071780 0x7f1d001578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cff400640 1 -- 172.21.15.92:0/2822018719 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076420886 0 0) 0x7f1d000c8a90 con 0x7f1d00071ed0 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cff400640 1 -- 172.21.15.92:0/2822018719 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ce4003520 con 0x7f1d00071ed0 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cffe00640 1 -- 172.21.15.92:0/2822018719 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 msgr2=0x7f1d00151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cffe00640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 0x7f1d00151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cffe00640 1 -- 172.21.15.92:0/2822018719 >> v1:172.21.15.92:6789/0 conn(0x7f1d00071ed0 legacy=0x7f1d000722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1cffe00640 1 -- 172.21.15.92:0/2822018719 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d000c3c60 con 0x7f1d00071780 2024-03-19T03:06:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.990+0000 7f1d05800640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 0x7f1d00151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.991+0000 7f1cffe00640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d00071780 0x7f1d001578f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0097d0 tx=0x7f1cec02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ded98ae3bf5c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.991+0000 7f1cff400640 1 -- 172.21.15.92:0/2822018719 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec034070 con 0x7f1d00071780 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.991+0000 7f1cff400640 1 -- 172.21.15.92:0/2822018719 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1cec030800 con 0x7f1d00071780 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.991+0000 7f1cff400640 1 -- 172.21.15.92:0/2822018719 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec030bf0 con 0x7f1d00071780 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 -- 172.21.15.92:0/2822018719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d00071780 msgr2=0x7f1d001578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d00071780 0x7f1d001578f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0097d0 tx=0x7f1cec02b940 comp rx=0 tx=0).stop 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 -- 172.21.15.92:0/2822018719 shutdown_connections 2024-03-19T03:06:44.994 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d000c8310 0x7f1d00151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.995 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 --2- 172.21.15.92:0/2822018719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d00071780 0x7f1d001578f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.995 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 -- 172.21.15.92:0/2822018719 >> 172.21.15.92:0/2822018719 conn(0x7f1d0006b240 msgr2=0x7f1d0006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.995 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 -- 172.21.15.92:0/2822018719 shutdown_connections 2024-03-19T03:06:44.995 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.992+0000 7f1d07800640 1 -- 172.21.15.92:0/2822018719 wait complete. 2024-03-19T03:06:44.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.994+0000 7f1d07800640 1 Processor -- start 2024-03-19T03:06:44.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.994+0000 7f1d07800640 1 -- start start 2024-03-19T03:06:44.997 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 0x7f1d0013b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 0x7f1d00136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 0x7f1d00137170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1d000c9530 con 0x7f1d000c8310 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1d000c9010 con 0x7f1d00071780 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d07800640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1d00137d20 con 0x7f1d00071ed0 2024-03-19T03:06:44.998 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d04e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 0x7f1d0013b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d05800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 0x7f1d00137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1cffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 0x7f1d00136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d04e00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 0x7f1d0013b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40592/0 (socket says 172.21.15.92:40592) 2024-03-19T03:06:45.001 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1cffe00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 0x7f1d00136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.92:52512/0 (socket says 172.21.15.92:52512) 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d04e00640 1 -- 172.21.15.92:0/308365707 learned_addr learned my addr 172.21.15.92:0/308365707 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d05800640 1 -- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 msgr2=0x7f1d00136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d05800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 0x7f1d00136c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d05800640 1 -- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 msgr2=0x7f1d0013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.995+0000 7f1d05800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 0x7f1d0013b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1d05800640 1 -- 172.21.15.92:0/308365707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0014e8c0 con 0x7f1d000c8310 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1d05800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 0x7f1d00137170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800efc0 tx=0x7f1cf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:45.002 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8018030 con 0x7f1d000c8310 2024-03-19T03:06:45.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1cf800cd30 con 0x7f1d000c8310 2024-03-19T03:06:45.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8003bd0 con 0x7f1d000c8310 2024-03-19T03:06:45.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d000caf50 con 0x7f1d000c8310 2024-03-19T03:06:45.003 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.996+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d00137fc0 con 0x7f1d000c8310 2024-03-19T03:06:45.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.997+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d000c4ce0 con 0x7f1d000c8310 2024-03-19T03:06:45.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.998+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1cf801b070 con 0x7f1d000c8310 2024-03-19T03:06:45.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.999+0000 7f1cfcc00640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 0x7f1ce006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.004 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.999+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6284+0+0 (secure 0 0 0) 0x7f1cf808b620 con 0x7f1d000c8310 2024-03-19T03:06:45.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:44.999+0000 7f1d04e00640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 0x7f1ce006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.000+0000 7f1d04e00640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 0x7f1ce006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0005880 tx=0x7f1cf0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:45.005 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.003+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1d000c4ce0 con 0x7f1d000c8310 2024-03-19T03:06:45.239 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.239+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1d000684c0 con 0x7f1d000c8310 2024-03-19T03:06:45.243 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.242+0000 7f1cfcc00640 1 -- 172.21.15.92:0/308365707 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f1cf8010040 con 0x7f1d000c8310 2024-03-19T03:06:45.243 INFO:teuthology.orchestra.run.smithi092.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T03:06:45.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 msgr2=0x7f1ce006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 0x7f1ce006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0005880 tx=0x7f1cf0009040 comp rx=0 tx=0).stop 2024-03-19T03:06:45.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 msgr2=0x7f1d00137170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.246 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 0x7f1d00137170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800efc0 tx=0x7f1cf800c490 comp rx=0 tx=0).stop 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 shutdown_connections 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f1ce006bbd0 0x7f1ce006e090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c8310 0x7f1d00137170 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d00071ed0 0x7f1d00136c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 --2- 172.21.15.92:0/308365707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d00071780 0x7f1d0013b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 >> 172.21.15.92:0/308365707 conn(0x7f1d0006b240 msgr2=0x7f1d0006e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 shutdown_connections 2024-03-19T03:06:45.247 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.245+0000 7f1d07800640 1 -- 172.21.15.92:0/308365707 wait complete. 2024-03-19T03:06:45.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T03:06:45.283 INFO:tasks.ceph.mds.f.smithi159.stdout:starting mds.f at 2024-03-19T03:06:45.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.460+0000 7f7a2ca00640 1 Processor -- start 2024-03-19T03:06:45.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.460+0000 7f7a2ca00640 1 -- start start 2024-03-19T03:06:45.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.461+0000 7f7a2ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7a28154eb0 0x7f7a280cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.461+0000 7f7a2ca00640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 0x7f7a280cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.461+0000 7f7a2ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a280ccbc0 con 0x7f7a280cbfa0 2024-03-19T03:06:45.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.461+0000 7f7a2ca00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2815e2d0 con 0x7f7a28154eb0 2024-03-19T03:06:45.463 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.461+0000 7f7a2ca00640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2815e5b0 con 0x7f7a280b36e0 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7a28154eb0 0x7f7a280cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7a28154eb0 0x7f7a280cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36922/0 (socket says 172.21.15.92:36922) 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 -- 172.21.15.92:0/1745395783 learned_addr learned my addr 172.21.15.92:0/1745395783 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27e00640 1 --1- 172.21.15.92:0/1745395783 >> v1:172.21.15.92:6789/0 conn(0x7f7a280cbfa0 0x7f7a280cc3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:44554/0 (socket says 172.21.15.92:44554) 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a1fe00640 1 --2- 172.21.15.92:0/1745395783 >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 0x7f7a280cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 -- 172.21.15.92:0/1745395783 >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 msgr2=0x7f7a280cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:45.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 --2- 172.21.15.92:0/1745395783 >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 0x7f7a280cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 -- 172.21.15.92:0/1745395783 >> v1:172.21.15.92:6789/0 conn(0x7f7a280cbfa0 legacy=0x7f7a280cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a26a00640 1 -- 172.21.15.92:0/1745395783 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1705444718 0 0) 0x7f7a280ccbc0 con 0x7f7a280cbfa0 2024-03-19T03:06:45.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a27400640 1 -- 172.21.15.92:0/1745395783 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2815e890 con 0x7f7a28154eb0 2024-03-19T03:06:45.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.462+0000 7f7a1fe00640 1 --2- 172.21.15.92:0/1745395783 >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 0x7f7a280cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.463+0000 7f7a27400640 1 --2- 172.21.15.92:0/1745395783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a28154eb0 0x7f7a280cb220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a280c74b0 tx=0x7f7a1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f23b4f425f8d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a26a00640 1 -- 172.21.15.92:0/1745395783 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18034070 con 0x7f7a28154eb0 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a26a00640 1 -- 172.21.15.92:0/1745395783 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7a18030860 con 0x7f7a28154eb0 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a26a00640 1 -- 172.21.15.92:0/1745395783 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18030c50 con 0x7f7a28154eb0 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 -- 172.21.15.92:0/1745395783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a28154eb0 msgr2=0x7f7a280cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/1745395783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a28154eb0 0x7f7a280cb220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a280c74b0 tx=0x7f7a1802b9a0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 -- 172.21.15.92:0/1745395783 shutdown_connections 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/1745395783 >> v2:172.21.15.159:3300/0 conn(0x7f7a280b36e0 0x7f7a280cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/1745395783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a28154eb0 0x7f7a280cb220 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 -- 172.21.15.92:0/1745395783 >> 172.21.15.92:0/1745395783 conn(0x7f7a28066f50 msgr2=0x7f7a28067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 -- 172.21.15.92:0/1745395783 shutdown_connections 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.464+0000 7f7a2ca00640 1 -- 172.21.15.92:0/1745395783 wait complete. 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.466+0000 7f7a2ca00640 1 Processor -- start 2024-03-19T03:06:45.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.466+0000 7f7a2ca00640 1 -- start start 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a280cbfa0 0x7f7a2815c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 0x7f7a2815cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7a28155290 con 0x7f7a280cbfa0 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2815ee50 con 0x7f7a280b36e0 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a2ca00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7a280ce210 con 0x7f7a28154eb0 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a27400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.471 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a280cbfa0 0x7f7a2815c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a27e00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 0x7f7a2815cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a27400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.92:40612/0 (socket says 172.21.15.92:40612) 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.467+0000 7f7a27400640 1 -- 172.21.15.92:0/892309686 learned_addr learned my addr 172.21.15.92:0/892309686 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 -- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 msgr2=0x7f7a2815cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 0x7f7a2815cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 -- 172.21.15.92:0/892309686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a280cbfa0 msgr2=0x7f7a2815c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a280cbfa0 0x7f7a2815c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:45.472 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 -- 172.21.15.92:0/892309686 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a281f1600 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27e00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 0x7f7a2815cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a27400640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18034040 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7a1803ab90 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18004030 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.468+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a280c66e0 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.469+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a280c6980 con 0x7f7a280b36e0 2024-03-19T03:06:45.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.470+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2804f580 con 0x7f7a280b36e0 2024-03-19T03:06:45.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.470+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a1804b060 con 0x7f7a280b36e0 2024-03-19T03:06:45.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.471+0000 7f7a1f400640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 0x7f7a0406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:45.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.471+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6370+0+0 (secure 0 0 0) 0x7f7a180acbc0 con 0x7f7a280b36e0 2024-03-19T03:06:45.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.471+0000 7f7a1fe00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 0x7f7a0406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:45.475 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.472+0000 7f7a1fe00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 0x7f7a0406e0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009560 tx=0x7f7a10009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:45.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.476+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a1803f070 con 0x7f7a280b36e0 2024-03-19T03:06:45.714 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:45.714+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7a280cbd80 con 0x7f7a280b36e0 2024-03-19T03:06:46.248 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.247+0000 7f7a1f400640 1 -- 172.21.15.92:0/892309686 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7f7a1807ad20 con 0x7f7a280b36e0 2024-03-19T03:06:46.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 msgr2=0x7f7a0406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 0x7f7a0406e0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009560 tx=0x7f7a10009190 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 msgr2=0x7f7a280cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a180046c0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 shutdown_connections 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f7a0406bc00 0x7f7a0406e0c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a28154eb0 0x7f7a2815cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a280cbfa0 0x7f7a2815c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 --2- 172.21.15.92:0/892309686 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a280b36e0 0x7f7a280cec60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 >> 172.21.15.92:0/892309686 conn(0x7f7a28066f50 msgr2=0x7f7a280cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:46.253 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 shutdown_connections 2024-03-19T03:06:46.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.251+0000 7f7a2ca00640 1 -- 172.21.15.92:0/892309686 wait complete. 2024-03-19T03:06:46.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T03:06:46.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.484+0000 7fb371600640 1 Processor -- start 2024-03-19T03:06:46.485 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.484+0000 7fb371600640 1 -- start start 2024-03-19T03:06:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.485+0000 7fb371600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb36c0cdcc0 0x7fb36c15ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.485+0000 7fb371600640 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb36c0cc230 0x7fb36c0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.485+0000 7fb371600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c0ccbe0 con 0x7fb36c154ec0 2024-03-19T03:06:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.485+0000 7fb371600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c15f000 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.485+0000 7fb371600640 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c1552a0 con 0x7fb36c0cc230 2024-03-19T03:06:46.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb36c0cdcc0 0x7fb36c15ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb36c0cdcc0 0x7fb36c15ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40062/0 (socket says 172.21.15.92:40062) 2024-03-19T03:06:46.487 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 -- 172.21.15.92:0/3644223222 learned_addr learned my addr 172.21.15.92:0/3644223222 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb36b400640 1 --2- 172.21.15.92:0/3644223222 >> v2:172.21.15.159:3300/0 conn(0x7fb36c0cc230 0x7fb36c0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 -- 172.21.15.92:0/3644223222 >> v2:172.21.15.159:3300/0 conn(0x7fb36c0cc230 msgr2=0x7fb36c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 --2- 172.21.15.92:0/3644223222 >> v2:172.21.15.159:3300/0 conn(0x7fb36c0cc230 0x7fb36c0cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 -- 172.21.15.92:0/3644223222 >> v1:172.21.15.92:6789/0 conn(0x7fb36c154ec0 legacy=0x7fb36c06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb362000640 1 -- 172.21.15.92:0/3644223222 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c15e480 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.488 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.486+0000 7fb36a000640 1 -- 172.21.15.92:0/3644223222 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291404725 0 0) 0x7fb36c0ccbe0 con 0x7fb36c154ec0 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.487+0000 7fb362000640 1 --2- 172.21.15.92:0/3644223222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c15ab50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb36c15bc50 tx=0x7fb35402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3163ca245749d203 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb36a000640 1 -- 172.21.15.92:0/3644223222 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354034070 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb36a000640 1 -- 172.21.15.92:0/3644223222 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb354030860 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb36a000640 1 -- 172.21.15.92:0/3644223222 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354030c50 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 -- 172.21.15.92:0/3644223222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 msgr2=0x7fb36c15ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 --2- 172.21.15.92:0/3644223222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c15ab50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb36c15bc50 tx=0x7fb35402b9a0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 -- 172.21.15.92:0/3644223222 shutdown_connections 2024-03-19T03:06:46.490 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 --2- 172.21.15.92:0/3644223222 >> v2:172.21.15.159:3300/0 conn(0x7fb36c0cc230 0x7fb36c0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 --2- 172.21.15.92:0/3644223222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c15ab50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 -- 172.21.15.92:0/3644223222 >> 172.21.15.92:0/3644223222 conn(0x7fb36c06da80 msgr2=0x7fb36c06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:46.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 -- 172.21.15.92:0/3644223222 shutdown_connections 2024-03-19T03:06:46.491 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.488+0000 7fb371600640 1 -- 172.21.15.92:0/3644223222 wait complete. 2024-03-19T03:06:46.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.490+0000 7fb371600640 1 Processor -- start 2024-03-19T03:06:46.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.490+0000 7fb371600640 1 -- start start 2024-03-19T03:06:46.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36c0cc230 0x7fb36c0b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c1f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.492 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 0x7fb36c1f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c15ed20 con 0x7fb36c0cdcc0 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c15ea40 con 0x7fb36c154ec0 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb371600640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c0cbd10 con 0x7fb36c0cc230 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb362000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c1f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb36b400640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 0x7fb36c1f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36c0cc230 0x7fb36c0b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb362000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c1f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40076/0 (socket says 172.21.15.92:40076) 2024-03-19T03:06:46.493 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.491+0000 7fb362000640 1 -- 172.21.15.92:0/1691166591 learned_addr learned my addr 172.21.15.92:0/1691166591 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 -- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36c0cc230 msgr2=0x7fb36c0b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36c0cc230 0x7fb36c0b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 -- 172.21.15.92:0/1691166591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 msgr2=0x7fb36c1f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c1f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 -- 172.21.15.92:0/1691166591 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c0b58c0 con 0x7fb36c154ec0 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb36b400640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 0x7fb36c1f14e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb364009d90 tx=0x7fb36400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.492+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364018020 con 0x7fb36c154ec0 2024-03-19T03:06:46.495 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.493+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb364014480 con 0x7fb36c154ec0 2024-03-19T03:06:46.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.493+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364014890 con 0x7fb36c154ec0 2024-03-19T03:06:46.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.493+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c0b4cb0 con 0x7fb36c154ec0 2024-03-19T03:06:46.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.493+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c0b5270 con 0x7fb36c154ec0 2024-03-19T03:06:46.496 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.494+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c0684c0 con 0x7fb36c154ec0 2024-03-19T03:06:46.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.495+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb364005d10 con 0x7fb36c154ec0 2024-03-19T03:06:46.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.496+0000 7fb363400640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:46.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.496+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6370+0+0 (secure 0 0 0) 0x7fb36408ba20 con 0x7fb36c154ec0 2024-03-19T03:06:46.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.496+0000 7fb36aa00640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:46.497 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.497+0000 7fb36aa00640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 0x7fb34806e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb35c000f60 tx=0x7fb35c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:46.501 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.501+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb364022030 con 0x7fb36c154ec0 2024-03-19T03:06:46.691 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:46.690+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb36c0b56a0 con 0x7fb36c154ec0 2024-03-19T03:06:47.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.391+0000 7fb363400640 1 -- 172.21.15.92:0/1691166591 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v40) ==== 181+0+0 (secure 0 0 0) 0x7fb364010040 con 0x7fb36c154ec0 2024-03-19T03:06:47.393 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_metadata' removed 2024-03-19T03:06:47.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.397+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 msgr2=0x7fb34806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.397+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 0x7fb34806e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb35c000f60 tx=0x7fb35c009040 comp rx=0 tx=0).stop 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.397+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 msgr2=0x7fb36c1f14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.397+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 0x7fb36c1f14e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb364009d90 tx=0x7fb36400c490 comp rx=0 tx=0).stop 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 shutdown_connections 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb36c154ec0 0x7fb36c1f14e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.399 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0cdcc0 0x7fb36c1f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 --2- 172.21.15.92:0/1691166591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb36c0cc230 0x7fb36c0b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.398+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 >> 172.21.15.92:0/1691166591 conn(0x7fb36c06da80 msgr2=0x7fb36c14fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:47.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.399+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 shutdown_connections 2024-03-19T03:06:47.400 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.399+0000 7fb371600640 1 -- 172.21.15.92:0/1691166591 wait complete. 2024-03-19T03:06:47.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T03:06:47.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.618+0000 7f4269000640 1 Processor -- start 2024-03-19T03:06:47.619 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.618+0000 7f4269000640 1 -- start start 2024-03-19T03:06:47.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4269000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 0x7f42640cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:47.620 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4269000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42640b3290 con 0x7f4264154de0 2024-03-19T03:06:47.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4269000640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42640b2cf0 con 0x7f42640b34c0 2024-03-19T03:06:47.621 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4269000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f42640cccc0 con 0x7f42640cd1c0 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4263400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4264154de0 0x7f42640cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:45726/0 (socket says 172.21.15.92:45726) 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4263400640 1 -- 172.21.15.92:0/4236067470 learned_addr learned my addr 172.21.15.92:0/4236067470 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.620+0000 7f4263e00640 1 --2- 172.21.15.92:0/4236067470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 0x7f42640cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608572185 0 0) 0x7f42640b3290 con 0x7f4264154de0 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4248003520 con 0x7f4264154de0 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3188080105 0 0) 0x7f42640b2cf0 con 0x7f42640b34c0 2024-03-19T03:06:47.622 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42640b3290 con 0x7f42640b34c0 2024-03-19T03:06:47.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4263e00640 1 -- 172.21.15.92:0/4236067470 >> v1:172.21.15.159:6789/0 conn(0x7f42640b34c0 legacy=0x7f426406bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4263e00640 1 -- 172.21.15.92:0/4236067470 >> v1:172.21.15.92:6789/0 conn(0x7f4264154de0 legacy=0x7f42640cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.621+0000 7f4263e00640 1 -- 172.21.15.92:0/4236067470 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426415ef90 con 0x7f42640cd1c0 2024-03-19T03:06:47.623 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.622+0000 7f4263e00640 1 --2- 172.21.15.92:0/4236067470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 0x7f42640cd630 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f425c009dc0 tx=0x7f425c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72cf00f29d541b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:47.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.623+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c039070 con 0x7f42640cd1c0 2024-03-19T03:06:47.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.623+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f425c02ccc0 con 0x7f42640cd1c0 2024-03-19T03:06:47.624 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.623+0000 7f4262000640 1 -- 172.21.15.92:0/4236067470 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c034640 con 0x7f42640cd1c0 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 -- 172.21.15.92:0/4236067470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 msgr2=0x7f42640cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 --2- 172.21.15.92:0/4236067470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 0x7f42640cd630 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f425c009dc0 tx=0x7f425c02bf10 comp rx=0 tx=0).stop 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 -- 172.21.15.92:0/4236067470 shutdown_connections 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 --2- 172.21.15.92:0/4236067470 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640cd1c0 0x7f42640cd630 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 -- 172.21.15.92:0/4236067470 >> 172.21.15.92:0/4236067470 conn(0x7f4264067130 msgr2=0x7f4264067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 -- 172.21.15.92:0/4236067470 shutdown_connections 2024-03-19T03:06:47.625 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.624+0000 7f4269000640 1 -- 172.21.15.92:0/4236067470 wait complete. 2024-03-19T03:06:47.626 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.626+0000 7f4269000640 1 Processor -- start 2024-03-19T03:06:47.627 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.626+0000 7f4269000640 1 -- start start 2024-03-19T03:06:47.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640b34c0 0x7f42640c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:47.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:47.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4264154de0 0x7f42640c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:47.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f426415e4b0 con 0x7f42640cd1c0 2024-03-19T03:06:47.628 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f426415e1d0 con 0x7f4264154de0 2024-03-19T03:06:47.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.627+0000 7f4269000640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f426415e790 con 0x7f42640b34c0 2024-03-19T03:06:47.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:47.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40084/0 (socket says 172.21.15.92:40084) 2024-03-19T03:06:47.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 -- 172.21.15.92:0/424322809 learned_addr learned my addr 172.21.15.92:0/424322809 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-19T03:06:47.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4263e00640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4264154de0 0x7f42640c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:47.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4263400640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640b34c0 0x7f42640c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 -- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640b34c0 msgr2=0x7f42640c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640b34c0 0x7f42640c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 -- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4264154de0 msgr2=0x7f42640c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4264154de0 0x7f42640c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.628+0000 7f4262a00640 1 -- 172.21.15.92:0/424322809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42640c8830 con 0x7f42640cd1c0 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.629+0000 7f4262a00640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f425000c490 tx=0x7f425000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.629+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42500052d0 con 0x7f42640cd1c0 2024-03-19T03:06:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.629+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4250005470 con 0x7f42640cd1c0 2024-03-19T03:06:47.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.629+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250012870 con 0x7f42640cd1c0 2024-03-19T03:06:47.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.629+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42640c82b0 con 0x7f42640cd1c0 2024-03-19T03:06:47.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.630+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426415b7e0 con 0x7f42640cd1c0 2024-03-19T03:06:47.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.631+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f426404f580 con 0x7f42640cd1c0 2024-03-19T03:06:47.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.632+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4250012a10 con 0x7f42640cd1c0 2024-03-19T03:06:47.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.632+0000 7f425b400640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 0x7f424406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:47.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.633+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5932+0+0 (secure 0 0 0) 0x7f425008a700 con 0x7f42640cd1c0 2024-03-19T03:06:47.633 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.633+0000 7f4263400640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 0x7f424406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:47.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.634+0000 7f4263400640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 0x7f424406e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f42540011e0 tx=0x7f4254007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:47.638 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.637+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4250058ab0 con 0x7f42640cd1c0 2024-03-19T03:06:47.838 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:47.836+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f42640cd640 con 0x7f42640cd1c0 2024-03-19T03:06:48.398 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.397+0000 7f425b400640 1 -- 172.21.15.92:0/424322809 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v41) ==== 169+0+0 (secure 0 0 0) 0x7f425005cb50 con 0x7f42640cd1c0 2024-03-19T03:06:48.398 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_data' removed 2024-03-19T03:06:48.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 msgr2=0x7f424406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:48.402 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 0x7f424406e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f42540011e0 tx=0x7f4254007d30 comp rx=0 tx=0).stop 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 msgr2=0x7f42640c5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f425000c490 tx=0x7f425000c950 comp rx=0 tx=0).stop 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 shutdown_connections 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:6832/697081936,v1:172.21.15.159:6833/697081936] conn(0x7f424406bbd0 0x7f424406e090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4264154de0 0x7f42640c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42640cd1c0 0x7f42640c5e40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:48.403 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 --2- 172.21.15.92:0/424322809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42640b34c0 0x7f42640c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:48.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 >> 172.21.15.92:0/424322809 conn(0x7f4264067130 msgr2=0x7f4264152fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:48.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 shutdown_connections 2024-03-19T03:06:48.404 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-19T03:06:48.401+0000 7f4269000640 1 -- 172.21.15.92:0/424322809 wait complete. 2024-03-19T03:06:48.434 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-19T03:06:48.434 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-19T03:06:48.434 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.457 INFO:tasks.ceph.mds.a:Stopped 2024-03-19T03:06:48.458 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-19T03:06:48.458 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.523 INFO:tasks.ceph.mds.c:Stopped 2024-03-19T03:06:48.523 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-03-19T03:06:48.523 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.579 INFO:tasks.ceph.mds.e:Stopped 2024-03-19T03:06:48.580 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-19T03:06:48.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.588 INFO:tasks.ceph.mds.b:Stopped 2024-03-19T03:06:48.589 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-19T03:06:48.589 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.638 INFO:tasks.ceph.mds.d:Stopped 2024-03-19T03:06:48.639 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-19T03:06:48.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.687 INFO:tasks.ceph.mds.f:Stopped 2024-03-19T03:06:48.687 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-19T03:06:48.688 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-19T03:06:48.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.784 INFO:tasks.ceph.osd.0:Stopped 2024-03-19T03:06:48.785 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-19T03:06:48.785 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.867 INFO:tasks.ceph.osd.1:Stopped 2024-03-19T03:06:48.867 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-19T03:06:48.867 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:48.958 INFO:tasks.ceph.osd.2:Stopped 2024-03-19T03:06:48.959 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-19T03:06:48.959 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.055 INFO:tasks.ceph.osd.3:Stopped 2024-03-19T03:06:49.055 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-19T03:06:49.056 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.093 INFO:tasks.ceph.osd.4:Stopped 2024-03-19T03:06:49.093 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-19T03:06:49.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.172 INFO:tasks.ceph.osd.5:Stopped 2024-03-19T03:06:49.173 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-19T03:06:49.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.242 INFO:tasks.ceph.osd.6:Stopped 2024-03-19T03:06:49.242 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-19T03:06:49.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.321 INFO:tasks.ceph.osd.7:Stopped 2024-03-19T03:06:49.321 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-19T03:06:49.322 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-19T03:06:49.322 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.408 INFO:tasks.ceph.mgr.y:Stopped 2024-03-19T03:06:49.408 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-19T03:06:49.408 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.435 INFO:tasks.ceph.mgr.x:Stopped 2024-03-19T03:06:49.435 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-19T03:06:49.436 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-19T03:06:49.436 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.471 INFO:tasks.ceph.mon.a:Stopped 2024-03-19T03:06:49.471 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-19T03:06:49.471 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.486 INFO:tasks.ceph.mon.b:Stopped 2024-03-19T03:06:49.486 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-19T03:06:49.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:06:49.536 INFO:tasks.ceph.mon.c:Stopped 2024-03-19T03:06:49.537 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-19T03:06:49.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | head -n 1 2024-03-19T03:06:49.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:06:49.665 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-19T03:06:50.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:06:50.172 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-19T03:06:50.322 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:06:50.322 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-19T03:06:50.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:06:50.470 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-19T03:06:50.633 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T03:06:50.634 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-19T03:06:52.315 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T03:06:52.315 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-19T03:06:52.393 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T03:06:52.394 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-19T03:06:52.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T03:06:52.450 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-19T03:06:52.527 INFO:tasks.ceph:Archiving mon data... 2024-03-19T03:06:52.528 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/data/mon.a.tgz 2024-03-19T03:06:52.528 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-19T03:06:52.585 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.zmHZyvDxf3 2024-03-19T03:06:52.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zmHZyvDxf3 2024-03-19T03:06:52.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.zmHZyvDxf3 2024-03-19T03:06:52.909 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.zmHZyvDxf3 is 322KB 2024-03-19T03:06:53.019 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.zmHZyvDxf3 2024-03-19T03:06:53.074 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/data/mon.b.tgz 2024-03-19T03:06:53.075 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-19T03:06:53.093 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.udiuvDY8f5 2024-03-19T03:06:53.093 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.udiuvDY8f5 2024-03-19T03:06:53.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.udiuvDY8f5 2024-03-19T03:06:53.324 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.udiuvDY8f5 is 334KB 2024-03-19T03:06:53.421 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.udiuvDY8f5 2024-03-19T03:06:53.435 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/data/mon.c.tgz 2024-03-19T03:06:53.436 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-19T03:06:53.490 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.yNNKFALhNr 2024-03-19T03:06:53.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yNNKFALhNr 2024-03-19T03:06:53.620 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.yNNKFALhNr 2024-03-19T03:06:53.705 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.yNNKFALhNr is 334KB 2024-03-19T03:06:53.801 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.yNNKFALhNr 2024-03-19T03:06:53.815 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-19T03:06:53.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T03:06:53.823 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T03:06:53.967 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T03:06:53.982 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T03:06:53.982 INFO:tasks.ceph:Archiving crash dumps... 2024-03-19T03:06:53.984 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi092/crash 2024-03-19T03:06:53.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T03:06:54.073 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi159/crash 2024-03-19T03:06:54.074 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T03:06:54.099 INFO:tasks.ceph:Shutting down logrotate 2024-03-19T03:06:54.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:54.107 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:54.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:54.333 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T03:06:54.361 INFO:tasks.ceph:Compressing logs... 2024-03-19T03:06:54.361 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:06:54.368 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:06:54.437 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-19T03:06:54.437 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68532.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68692.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68532.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68692.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68532.log.gz 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68692.log.gz 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68612.log 2024-03-19T03:06:54.438 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68732.log 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68612.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68732.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68612.log.gz 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.7.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68732.log.gz 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68572.log 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T03:06:54.439 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68492.log 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mds.f.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68572.log.gz 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose 93.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr: -- /var/log/ceph/ceph-mgr.x.log 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68652.log 2024-03-19T03:06:54.440 INFO:teuthology.orchestra.run.smithi159.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68492.log.gz 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68652.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68652.log.gz 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mon.b.log: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T03:06:54.441 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68452.log 2024-03-19T03:06:54.451 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-19T03:06:54.451 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.68452.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68452.log.gz 2024-03-19T03:06:54.453 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42450.log 2024-03-19T03:06:54.454 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2024-03-19T03:06:54.455 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2024-03-19T03:06:54.455 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42450.log.gz 2024-03-19T03:06:54.456 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53067.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2024-03-19T03:06:54.457 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51894.log 2024-03-19T03:06:54.457 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47399.log 2024-03-19T03:06:54.458 INFO:teuthology.orchestra.run.smithi092.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2024-03-19T03:06:54.458 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51414.log 2024-03-19T03:06:54.459 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51894.log.gz 2024-03-19T03:06:54.460 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47399.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-19T03:06:54.460 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47399.log.gz 2024-03-19T03:06:54.461 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51414.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.51414.log.gz /var/log/ceph/ceph-client.admin.52130.log 2024-03-19T03:06:54.461 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-19T03:06:54.462 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43536.log 2024-03-19T03:06:54.463 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52130.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2024-03-19T03:06:54.463 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52298.log 2024-03-19T03:06:54.464 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.0.43536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43536.log.gz 2024-03-19T03:06:54.464 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-03-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50283.log 2024-03-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52298.log.gz 2024-03-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2024-03-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi092.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-03-19T03:06:54.467 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-19T03:06:54.467 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50283.log.gz 2024-03-19T03:06:54.468 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46050.log 2024-03-19T03:06:54.468 INFO:teuthology.orchestra.run.smithi092.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2024-03-19T03:06:54.469 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52857.log 2024-03-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.46050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49695.log 2024-03-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46050.log.gz 2024-03-19T03:06:54.471 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53671.log 2024-03-19T03:06:54.471 INFO:teuthology.orchestra.run.smithi092.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52857.log.gz 2024-03-19T03:06:54.472 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-03-19T03:06:54.473 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49695.log.gz 2024-03-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53671.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53671.log.gz 2024-03-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44341.log 2024-03-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53025.log 2024-03-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-03-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50763.log 2024-03-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44341.log.gz 2024-03-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50511.log 2024-03-19T03:06:54.477 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53025.log.gz 2024-03-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53151.log 2024-03-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50763.log.gz 2024-03-19T03:06:54.479 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52214.log 2024-03-19T03:06:54.479 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50511.log.gz 2024-03-19T03:06:54.480 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36917.log 2024-03-19T03:06:54.480 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2024-03-19T03:06:54.481 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37275.log 2024-03-19T03:06:54.481 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52214.log.gz 2024-03-19T03:06:54.482 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.36917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47483.log 2024-03-19T03:06:54.482 INFO:teuthology.orchestra.run.smithi092.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36917.log.gz 2024-03-19T03:06:54.483 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49478.log 2024-03-19T03:06:54.483 INFO:teuthology.orchestra.run.smithi092.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37275.log.gz 2024-03-19T03:06:54.484 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45948.log 2024-03-19T03:06:54.484 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47483.log.gz 2024-03-19T03:06:54.485 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2024-03-19T03:06:54.485 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49478.log.gz 2024-03-19T03:06:54.486 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52983.log 2024-03-19T03:06:54.486 INFO:teuthology.orchestra.run.smithi092.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45948.log.gz 2024-03-19T03:06:54.487 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52815.log 2024-03-19T03:06:54.487 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2024-03-19T03:06:54.488 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51978.log 2024-03-19T03:06:54.488 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52983.log.gz 2024-03-19T03:06:54.489 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45990.log 2024-03-19T03:06:54.489 INFO:teuthology.orchestra.run.smithi092.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52815.log.gz 2024-03-19T03:06:54.490 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42534.log 2024-03-19T03:06:54.491 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51978.log.gz 2024-03-19T03:06:54.492 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41907.log 2024-03-19T03:06:54.492 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45990.log.gz 2024-03-19T03:06:54.493 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42492.log 2024-03-19T03:06:54.493 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42534.log.gz 2024-03-19T03:06:54.494 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50721.log 2024-03-19T03:06:54.494 INFO:teuthology.orchestra.run.smithi092.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41907.log.gz 2024-03-19T03:06:54.495 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2024-03-19T03:06:54.495 INFO:teuthology.orchestra.run.smithi092.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42492.log.gz 2024-03-19T03:06:54.496 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43711.log 2024-03-19T03:06:54.496 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50721.log.gz 2024-03-19T03:06:54.498 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51624.log 2024-03-19T03:06:54.499 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-03-19T03:06:54.499 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49604.log 2024-03-19T03:06:54.499 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43711.log.gz 2024-03-19T03:06:54.499 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40873.log 2024-03-19T03:06:54.499 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51624.log.gz 2024-03-19T03:06:54.500 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52899.log 2024-03-19T03:06:54.500 INFO:teuthology.orchestra.run.smithi092.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49604.log.gz 2024-03-19T03:06:54.501 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.40873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50949.log 2024-03-19T03:06:54.501 INFO:teuthology.orchestra.run.smithi092.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40873.log.gz 2024-03-19T03:06:54.502 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-03-19T03:06:54.502 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52899.log.gz 2024-03-19T03:06:54.503 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-19T03:06:54.503 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50949.log.gz 2024-03-19T03:06:54.504 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50240.log 2024-03-19T03:06:54.504 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2024-03-19T03:06:54.504 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49024.log 2024-03-19T03:06:54.505 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-19T03:06:54.506 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41298.log 2024-03-19T03:06:54.506 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50240.log.gz 2024-03-19T03:06:54.507 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-19T03:06:54.507 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49024.log.gz 2024-03-19T03:06:54.507 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47441.log 2024-03-19T03:06:54.508 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41298.log.gz 2024-03-19T03:06:54.509 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2024-03-19T03:06:54.509 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-19T03:06:54.510 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2024-03-19T03:06:54.510 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47441.log.gz 2024-03-19T03:06:54.511 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52647.log 2024-03-19T03:06:54.511 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2024-03-19T03:06:54.512 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49295.log 2024-03-19T03:06:54.512 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2024-03-19T03:06:54.512 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43963.log 2024-03-19T03:06:54.513 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52647.log.gz 2024-03-19T03:06:54.514 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45300.log 2024-03-19T03:06:54.514 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49295.log.gz 2024-03-19T03:06:54.515 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2024-03-19T03:06:54.515 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43963.log.gz 2024-03-19T03:06:54.516 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2024-03-19T03:06:54.516 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45300.log.gz 2024-03-19T03:06:54.517 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-03-19T03:06:54.517 INFO:teuthology.orchestra.run.smithi092.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2024-03-19T03:06:54.518 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47609.log 2024-03-19T03:06:54.518 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2024-03-19T03:06:54.519 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41916.log 2024-03-19T03:06:54.519 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-03-19T03:06:54.520 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47567.log 2024-03-19T03:06:54.520 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47609.log.gz 2024-03-19T03:06:54.521 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48940.log 2024-03-19T03:06:54.521 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41916.log.gz 2024-03-19T03:06:54.522 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41162.log 2024-03-19T03:06:54.522 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47567.log.gz 2024-03-19T03:06:54.523 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2024-03-19T03:06:54.523 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48940.log.gz 2024-03-19T03:06:54.524 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-03-19T03:06:54.524 INFO:teuthology.orchestra.run.smithi092.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41162.log.gz 2024-03-19T03:06:54.525 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43921.log 2024-03-19T03:06:54.525 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2024-03-19T03:06:54.526 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-03-19T03:06:54.526 INFO:teuthology.orchestra.run.smithi092.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-03-19T03:06:54.527 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45906.log 2024-03-19T03:06:54.527 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43921.log.gz 2024-03-19T03:06:54.528 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2024-03-19T03:06:54.528 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-03-19T03:06:54.529 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-03-19T03:06:54.529 INFO:teuthology.orchestra.run.smithi092.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45906.log.gz 2024-03-19T03:06:54.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2024-03-19T03:06:54.530 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2024-03-19T03:06:54.531 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44257.log 2024-03-19T03:06:54.531 INFO:teuthology.orchestra.run.smithi092.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-03-19T03:06:54.532 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53109.log 2024-03-19T03:06:54.532 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2024-03-19T03:06:54.533 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44257.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T03:06:54.533 INFO:teuthology.orchestra.run.smithi092.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44257.log.gz 2024-03-19T03:06:54.534 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43669.log 2024-03-19T03:06:54.534 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53109.log.gz 2024-03-19T03:06:54.535 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48492.log 2024-03-19T03:06:54.536 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43669.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T03:06:54.536 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-03-19T03:06:54.536 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43669.log.gz 2024-03-19T03:06:54.537 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2024-03-19T03:06:54.537 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48492.log.gz 2024-03-19T03:06:54.538 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51936.log 2024-03-19T03:06:54.538 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-03-19T03:06:54.539 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.40659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41914.log 2024-03-19T03:06:54.539 INFO:teuthology.orchestra.run.smithi092.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2024-03-19T03:06:54.540 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2024-03-19T03:06:54.540 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51936.log.gz 2024-03-19T03:06:54.541 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51177.log 2024-03-19T03:06:54.541 INFO:teuthology.orchestra.run.smithi092.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41914.log.gz 2024-03-19T03:06:54.542 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49520.log 2024-03-19T03:06:54.542 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2024-03-19T03:06:54.543 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52256.log 2024-03-19T03:06:54.543 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51177.log.gz 2024-03-19T03:06:54.544 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52479.log 2024-03-19T03:06:54.544 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49520.log.gz 2024-03-19T03:06:54.545 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43038.log 2024-03-19T03:06:54.545 INFO:teuthology.orchestra.run.smithi092.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52256.log.gz 2024-03-19T03:06:54.546 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47273.log 2024-03-19T03:06:54.546 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52479.log.gz 2024-03-19T03:06:54.547 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53235.log 2024-03-19T03:06:54.547 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2024-03-19T03:06:54.548 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50114.log 2024-03-19T03:06:54.548 INFO:teuthology.orchestra.run.smithi092.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47273.log.gz 2024-03-19T03:06:54.549 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53503.log 2024-03-19T03:06:54.549 INFO:teuthology.orchestra.run.smithi092.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53235.log.gz 2024-03-19T03:06:54.550 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51708.log 2024-03-19T03:06:54.550 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50114.log.gz 2024-03-19T03:06:54.551 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-03-19T03:06:54.551 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53503.log.gz 2024-03-19T03:06:54.552 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47357.log 2024-03-19T03:06:54.552 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51708.log.gz 2024-03-19T03:06:54.553 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-03-19T03:06:54.553 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-03-19T03:06:54.554 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-03-19T03:06:54.555 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47357.log.gz 2024-03-19T03:06:54.555 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51372.log 2024-03-19T03:06:54.556 INFO:teuthology.orchestra.run.smithi092.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-03-19T03:06:54.556 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2024-03-19T03:06:54.556 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-03-19T03:06:54.557 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2024-03-19T03:06:54.558 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51372.log.gz 2024-03-19T03:06:54.558 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-19T03:06:54.559 INFO:teuthology.orchestra.run.smithi092.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2024-03-19T03:06:54.560 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41917.log 2024-03-19T03:06:54.560 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2024-03-19T03:06:54.560 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49235.log 2024-03-19T03:06:54.561 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45612.log 2024-03-19T03:06:54.562 INFO:teuthology.orchestra.run.smithi092.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41917.log.gz 2024-03-19T03:06:54.562 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49235.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49235.log.gz 2024-03-19T03:06:54.563 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53587.log 2024-03-19T03:06:54.564 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45612.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30902.log 2024-03-19T03:06:54.564 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45612.log.gz 2024-03-19T03:06:54.565 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2024-03-19T03:06:54.565 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53587.log.gz 2024-03-19T03:06:54.565 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.tmp-client.admin.30902.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30902.log.gz 2024-03-19T03:06:54.566 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52731.log 2024-03-19T03:06:54.567 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2024-03-19T03:06:54.567 INFO:teuthology.orchestra.run.smithi092.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2024-03-19T03:06:54.568 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2024-03-19T03:06:54.568 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52731.log.gz 2024-03-19T03:06:54.569 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52689.log 2024-03-19T03:06:54.570 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2024-03-19T03:06:54.571 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2024-03-19T03:06:54.571 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44047.log 2024-03-19T03:06:54.572 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51135.log 2024-03-19T03:06:54.572 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52689.log.gz 2024-03-19T03:06:54.573 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51810.log 2024-03-19T03:06:54.573 INFO:teuthology.orchestra.run.smithi092.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44047.log.gz 2024-03-19T03:06:54.574 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2024-03-19T03:06:54.574 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51135.log.gz 2024-03-19T03:06:54.575 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52172.log 2024-03-19T03:06:54.576 INFO:teuthology.orchestra.run.smithi092.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51810.log.gz 2024-03-19T03:06:54.576 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48363.log 2024-03-19T03:06:54.577 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2024-03-19T03:06:54.578 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51498.log 2024-03-19T03:06:54.578 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52172.log.gz 2024-03-19T03:06:54.579 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2024-03-19T03:06:54.579 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48363.log.gz 2024-03-19T03:06:54.580 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51288.log 2024-03-19T03:06:54.580 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51498.log.gz 2024-03-19T03:06:54.581 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42828.log 2024-03-19T03:06:54.581 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz 2024-03-19T03:06:54.582 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50072.log 2024-03-19T03:06:54.582 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51288.log.gz 2024-03-19T03:06:54.583 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-03-19T03:06:54.583 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42828.log.gz 2024-03-19T03:06:54.584 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45738.log 2024-03-19T03:06:54.585 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50072.log.gz 2024-03-19T03:06:54.585 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42576.log 2024-03-19T03:06:54.586 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-03-19T03:06:54.587 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45738.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45738.log.gz -5 2024-03-19T03:06:54.587 INFO:teuthology.orchestra.run.smithi092.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44215.log 2024-03-19T03:06:54.588 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-03-19T03:06:54.588 INFO:teuthology.orchestra.run.smithi092.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42576.log.gz 2024-03-19T03:06:54.589 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44215.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-19T03:06:54.589 INFO:teuthology.orchestra.run.smithi092.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44215.log.gz 2024-03-19T03:06:54.591 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49394.log 2024-03-19T03:06:54.591 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-03-19T03:06:54.592 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2024-03-19T03:06:54.592 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49394.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49394.log.gz 2024-03-19T03:06:54.593 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51852.log 2024-03-19T03:06:54.594 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42248.log 2024-03-19T03:06:54.594 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44299.log: /var/log/ceph/ceph-client.admin.51852.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2024-03-19T03:06:54.595 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51852.log.gz 2024-03-19T03:06:54.595 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43517.log 2024-03-19T03:06:54.596 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52941.log 2024-03-19T03:06:54.596 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42248.log.gz 2024-03-19T03:06:54.597 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.0.43517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43517.log.gz 2024-03-19T03:06:54.597 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42786.log 2024-03-19T03:06:54.598 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51246.log 2024-03-19T03:06:54.599 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52941.log: /var/log/ceph/ceph-client.admin.42786.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52941.log.gz 2024-03-19T03:06:54.599 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42786.log.gz 2024-03-19T03:06:54.599 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46092.log 2024-03-19T03:06:54.600 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52773.log 2024-03-19T03:06:54.600 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51246.log.gz 2024-03-19T03:06:54.601 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.46092.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46092.log.gz 2024-03-19T03:06:54.602 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47315.log 2024-03-19T03:06:54.603 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43795.log 2024-03-19T03:06:54.604 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52773.log: /var/log/ceph/ceph-client.admin.47315.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52773.log.gz 2024-03-19T03:06:54.604 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47315.log.gz 2024-03-19T03:06:54.604 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43879.log 2024-03-19T03:06:54.605 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53277.log 2024-03-19T03:06:54.606 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43795.log.gz 2024-03-19T03:06:54.606 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43879.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43879.log.gz 2024-03-19T03:06:54.606 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2024-03-19T03:06:54.607 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2024-03-19T03:06:54.608 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53277.log: /var/log/ceph/ceph-client.admin.44131.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53277.log.gz 2024-03-19T03:06:54.608 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2024-03-19T03:06:54.609 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-03-19T03:06:54.609 INFO:teuthology.orchestra.run.smithi092.stderr:gzip/var/log/ceph/ceph-client.admin.50325.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42249.log 2024-03-19T03:06:54.610 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41736.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2024-03-19T03:06:54.611 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-03-19T03:06:54.611 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2024-03-19T03:06:54.612 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41214.log 2024-03-19T03:06:54.612 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42249.log: /var/log/ceph/ceph-client.admin.50409.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42249.log.gz 2024-03-19T03:06:54.613 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2024-03-19T03:06:54.613 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49988.log 2024-03-19T03:06:54.614 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41214.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-19T03:06:54.614 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41214.log.gz 2024-03-19T03:06:54.615 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49988.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49988.log.gz 2024-03-19T03:06:54.615 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50637.log 2024-03-19T03:06:54.616 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45654.log 2024-03-19T03:06:54.617 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.50637.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50637.log.gz 2024-03-19T03:06:54.617 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53419.log 2024-03-19T03:06:54.618 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45654.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-19T03:06:54.623 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45654.log.gz 2024-03-19T03:06:54.623 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53419.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53419.log.gz 2024-03-19T03:06:54.623 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2024-03-19T03:06:54.623 INFO:teuthology.orchestra.run.smithi092.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-19T03:06:54.623 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45864.log 2024-03-19T03:06:54.624 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 88.6% 2024-03-19T03:06:54.624 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz 2024-03-19T03:06:54.624 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45864.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45864.log.gz 2024-03-19T03:06:54.625 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45486.log 2024-03-19T03:06:54.625 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43182.log 2024-03-19T03:06:54.625 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-03-19T03:06:54.626 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47837.log 2024-03-19T03:06:54.626 INFO:teuthology.orchestra.run.smithi092.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45486.log.gz 2024-03-19T03:06:54.626 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52437.log 2024-03-19T03:06:54.627 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43182.log.gz 2024-03-19T03:06:54.628 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49109.log 2024-03-19T03:06:54.628 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47837.log.gz 2024-03-19T03:06:54.629 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52437.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52437.log.gzgzip 2024-03-19T03:06:54.629 INFO:teuthology.orchestra.run.smithi092.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50156.log 2024-03-19T03:06:54.630 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40774.log 2024-03-19T03:06:54.630 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49109.log.gz 2024-03-19T03:06:54.631 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47525.log 2024-03-19T03:06:54.631 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50156.log.gz 2024-03-19T03:06:54.633 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.40774.log: gzip -5 88.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40774.log.gz -- 2024-03-19T03:06:54.633 INFO:teuthology.orchestra.run.smithi092.stderr: /var/log/ceph/ceph-client.admin.50198.log 2024-03-19T03:06:54.633 INFO:teuthology.orchestra.run.smithi092.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-19T03:06:54.633 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49646.log 2024-03-19T03:06:54.634 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50198.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47525.log.gz 2024-03-19T03:06:54.635 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 87.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz -- 2024-03-19T03:06:54.635 INFO:teuthology.orchestra.run.smithi092.stderr: /var/log/ceph/ceph-client.admin.51750.log 2024-03-19T03:06:54.636 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49646.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49646.log.gz 2024-03-19T03:06:54.636 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40960.log 2024-03-19T03:06:54.637 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44173.log 2024-03-19T03:06:54.637 INFO:teuthology.orchestra.run.smithi092.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51750.log.gz 2024-03-19T03:06:54.638 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.40960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-03-19T03:06:54.638 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-03-19T03:06:54.639 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44173.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44173.log.gz 2024-03-19T03:06:54.640 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42912.log 2024-03-19T03:06:54.640 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41909.log 2024-03-19T03:06:54.641 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-03-19T03:06:54.642 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50847.log 2024-03-19T03:06:54.642 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42912.log.gz 2024-03-19T03:06:54.643 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42996.log 2024-03-19T03:06:54.643 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41909.log.gz 2024-03-19T03:06:54.644 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51582.log 2024-03-19T03:06:54.644 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2%/var/log/ceph/ceph-client.admin.42996.log: -- replaced with /var/log/ceph/ceph-client.admin.50847.log.gz 2024-03-19T03:06:54.645 INFO:teuthology.orchestra.run.smithi092.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42996.log.gz 2024-03-19T03:06:54.645 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49337.log 2024-03-19T03:06:54.647 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51582.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51582.log.gz 2024-03-19T03:06:54.647 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52020.log 2024-03-19T03:06:54.647 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49337.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-19T03:06:54.648 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49337.log.gz 2024-03-19T03:06:54.649 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52020.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52020.log.gz 2024-03-19T03:06:54.649 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2024-03-19T03:06:54.650 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2024-03-19T03:06:54.651 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49738.log 2024-03-19T03:06:54.651 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2024-03-19T03:06:54.653 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53545.log 2024-03-19T03:06:54.653 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2024-03-19T03:06:54.654 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2024-03-19T03:06:54.654 INFO:teuthology.orchestra.run.smithi092.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49738.log.gz 2024-03-19T03:06:54.655 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53629.log 2024-03-19T03:06:54.656 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53545.log.gz 2024-03-19T03:06:54.656 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mon.c.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-19T03:06:54.657 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49193.log 2024-03-19T03:06:54.657 INFO:teuthology.orchestra.run.smithi092.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2024-03-19T03:06:54.658 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50889.log 2024-03-19T03:06:54.658 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53629.log.gz 2024-03-19T03:06:54.659 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42254.log 2024-03-19T03:06:54.659 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49193.log.gz 2024-03-19T03:06:54.660 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2024-03-19T03:06:54.661 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50889.log.gz 2024-03-19T03:06:54.662 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-03-19T03:06:54.662 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42254.log.gz 2024-03-19T03:06:54.663 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51051.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T03:06:54.663 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2024-03-19T03:06:54.664 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49436.log 2024-03-19T03:06:54.664 INFO:teuthology.orchestra.run.smithi092.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-03-19T03:06:54.665 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-19T03:06:54.667 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49436.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-19T03:06:54.667 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49436.log.gz 2024-03-19T03:06:54.668 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51666.log 2024-03-19T03:06:54.668 INFO:teuthology.orchestra.run.smithi092.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T03:06:54.668 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-19T03:06:54.669 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42618.log 2024-03-19T03:06:54.671 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41568.log 2024-03-19T03:06:54.671 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51666.log.gz 2024-03-19T03:06:54.672 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42618.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42618.log.gz 2024-03-19T03:06:54.672 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-19T03:06:54.673 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41568.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41568.log.gz 2024-03-19T03:06:54.673 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41610.log 2024-03-19T03:06:54.675 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45780.log 2024-03-19T03:06:54.676 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53377.log 2024-03-19T03:06:54.677 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41610.log.gz 2024-03-19T03:06:54.678 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45780.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45780.log.gz 2024-03-19T03:06:54.678 INFO:teuthology.orchestra.run.smithi092.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-19T03:06:54.678 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42251.log 2024-03-19T03:06:54.679 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53377.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53377.log.gz 2024-03-19T03:06:54.681 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42251.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42251.log.gz 2024-03-19T03:06:54.682 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41911.log 2024-03-19T03:06:54.683 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2024-03-19T03:06:54.684 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41911.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52563.log 2024-03-19T03:06:54.685 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41911.log.gz 2024-03-19T03:06:54.686 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2024-03-19T03:06:54.686 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz 2024-03-19T03:06:54.687 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52563.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52563.log.gz 2024-03-19T03:06:54.688 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44005.log 2024-03-19T03:06:54.689 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47963.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz -5 2024-03-19T03:06:54.689 INFO:teuthology.orchestra.run.smithi092.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-03-19T03:06:54.690 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.44005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43080.log 2024-03-19T03:06:54.690 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44005.log.gz 2024-03-19T03:06:54.692 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41442.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-03-19T03:06:54.693 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5/var/log/ceph/ceph-client.admin.43080.log: --verbose -- /var/log/ceph/ceph-client.admin.43753.log 2024-03-19T03:06:54.693 INFO:teuthology.orchestra.run.smithi092.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43080.log.gz 2024-03-19T03:06:54.694 INFO:teuthology.orchestra.run.smithi159.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-19T03:06:54.694 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45384.log 2024-03-19T03:06:54.695 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.43753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42660.log 2024-03-19T03:06:54.695 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43753.log.gz 2024-03-19T03:06:54.697 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45384.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45384.log.gz 2024-03-19T03:06:54.698 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42660.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42660.log.gz 2024-03-19T03:06:54.698 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49562.log 2024-03-19T03:06:54.699 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53193.log 2024-03-19T03:06:54.701 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49151.log 2024-03-19T03:06:54.701 INFO:teuthology.orchestra.run.smithi092.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49562.log.gz 2024-03-19T03:06:54.702 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53193.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53193.log.gz -5 2024-03-19T03:06:54.703 INFO:teuthology.orchestra.run.smithi092.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2024-03-19T03:06:54.704 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.49151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42870.log 2024-03-19T03:06:54.704 INFO:teuthology.orchestra.run.smithi092.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49151.log.gz 2024-03-19T03:06:54.705 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.53461.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2024-03-19T03:06:54.706 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51540.log 2024-03-19T03:06:54.706 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42870.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42870.log.gz 2024-03-19T03:06:54.707 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2024-03-19T03:06:54.709 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48982.log 2024-03-19T03:06:54.709 INFO:teuthology.orchestra.run.smithi092.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51540.log.gz 2024-03-19T03:06:54.710 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50595.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2024-03-19T03:06:54.710 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46159.log 2024-03-19T03:06:54.712 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.48982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-03-19T03:06:54.713 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48982.log.gz/var/log/ceph/ceph-client.admin.46159.log: 2024-03-19T03:06:54.713 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46159.log.gz 2024-03-19T03:06:54.713 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-19T03:06:54.714 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.41484.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-03-19T03:06:54.715 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50030.log 2024-03-19T03:06:54.716 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50805.log 2024-03-19T03:06:54.717 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.50030.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50030.log.gz 2024-03-19T03:06:54.719 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5/var/log/ceph/ceph-client.admin.50805.log: --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-03-19T03:06:54.719 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50805.log.gz 2024-03-19T03:06:54.720 INFO:teuthology.orchestra.run.smithi092.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-19T03:06:54.720 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42702.log 2024-03-19T03:06:54.721 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.51330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45528.log 2024-03-19T03:06:54.722 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-03-19T03:06:54.722 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.42702.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42702.log.gz 2024-03-19T03:06:54.723 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-03-19T03:06:54.724 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.45528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52605.log 2024-03-19T03:06:54.724 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45528.log.gz 2024-03-19T03:06:54.725 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.47879.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-03-19T03:06:54.727 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.52605.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52605.log.gz 2024-03-19T03:06:54.904 INFO:teuthology.orchestra.run.smithi159.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-19T03:06:54.958 INFO:teuthology.orchestra.run.smithi092.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-19T03:06:58.125 INFO:teuthology.orchestra.run.smithi092.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-19T03:06:58.770 INFO:teuthology.orchestra.run.smithi092.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-19T03:06:59.567 INFO:teuthology.orchestra.run.smithi092.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-19T03:06:59.794 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-19T03:06:59.798 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-19T03:06:59.798 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m5.384s 2024-03-19T03:06:59.799 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m17.099s 2024-03-19T03:06:59.799 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m1.935s 2024-03-19T03:07:00.550 INFO:teuthology.orchestra.run.smithi159.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-19T03:07:00.722 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-19T03:07:02.586 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-19T03:07:02.733 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-19T03:07:10.888 INFO:teuthology.orchestra.run.smithi159.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-19T03:07:20.094 INFO:teuthology.orchestra.run.smithi159.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-19T03:07:20.095 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-19T03:07:20.095 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m25.680s 2024-03-19T03:07:20.096 INFO:teuthology.orchestra.run.smithi159.stderr:user 1m9.360s 2024-03-19T03:07:20.096 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m2.180s 2024-03-19T03:07:20.096 INFO:tasks.ceph:Archiving logs... 2024-03-19T03:07:20.097 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi092/log 2024-03-19T03:07:20.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:07:23.045 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi159/log 2024-03-19T03:07:23.046 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:07:32.346 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T03:07:32.357 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T03:07:32.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:07:32.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:07:32.509 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T03:07:32.519 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T03:07:32.519 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:07:32.527 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:07:32.546 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +2911us[+2911us] +/- 108ms 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv02.front.sepia.ceph.com 3 7 377 118 -1347us[-1376us] +/- 81ms 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -437us[ -437us] +/- 74ms 2024-03-19T03:07:32.551 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:07:32.571 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +3026us[+2793us] +/- 108ms 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -1306us[-1539us] +/- 49ms 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -355us[ -355us] +/- 74ms 2024-03-19T03:07:32.581 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:07:32.583 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T03:07:32.593 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T03:07:32.593 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T03:07:32.626 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:07:32.846 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:07:32.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:07:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:33.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:07:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:33.051 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T03:07:33.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T03:07:33.093 INFO:teuthology.task.internal:Duration was 1913.822262 seconds 2024-03-19T03:07:33.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T03:07:33.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T03:07:33.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:07:33.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:07:33.199 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:07:33.298 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:07:33.764 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T03:07:33.765 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-03-19T03:07:33.765 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:07:33.830 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-03-19T03:07:33.830 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:07:33.860 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T03:07:33.861 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:07:33.865 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:07:33.953 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T03:07:33.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:07:33.959 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:07:34.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T03:07:34.156 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T03:07:34.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:07:34.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:07:34.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T03:07:34.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:34.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:34.346 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-03-19T03:07:34.401 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-03-19T03:07:34.452 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:34.501 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:34.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T03:07:34.527 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T03:07:34.528 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi092 2024-03-19T03:07:34.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:07:34.635 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441/remote/smithi159 2024-03-19T03:07:34.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:07:34.685 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T03:07:34.686 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:07:34.693 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:07:34.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T03:07:34.754 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T03:07:34.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T03:07:34.795 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T03:07:34.796 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:07:34.805 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:07:34.819 INFO:teuthology.orchestra.run.smithi159.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:07 /home/ubuntu/cephtest 2024-03-19T03:07:34.851 INFO:teuthology.orchestra.run.smithi092.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:07 /home/ubuntu/cephtest 2024-03-19T03:07:34.854 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T03:07:34.868 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T03:07:34.961 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T03:07:34.985 DEBUG:teuthology.nuke:shortname: smithi092 2024-03-19T03:07:34.986 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:07:35.007 DEBUG:teuthology.nuke:shortname: smithi159 2024-03-19T03:07:35.007 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:07:35.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.858958', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFx719IkU9FJAu59YtRWy7dpjI33NS5tXulL6SGSomLxamg9kPlqlUVUqWY0X+kfcWkNNlTabPYR7Zi+Zj+FNo='} 2024-03-19T03:07:35.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.860582', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDa8khp7bshqgwJj1KEBwKqaixzbd4GAw47cz1EWzErQ2TUConfFiFw4/FY8kSd10LsmnQXHr5Lw0wyvVfMeYkA='} 2024-03-19T03:07:35.060 INFO:teuthology.orchestra.console.smithi092:Power off 2024-03-19T03:07:35.060 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:07:35.085 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-03-19T03:07:35.085 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:35.103 INFO:teuthology.orchestra.console.smithi159:Power off 2024-03-19T03:07:35.104 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:07:35.116 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-19T03:07:35.134 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2024-03-19T03:07:35.134 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:35.160 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-19T03:07:39.117 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:39.161 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:39.228 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-19T03:07:39.272 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-19T03:07:43.230 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:43.274 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:43.342 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-03-19T03:07:43.384 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2024-03-19T03:07:43.443 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-03-19T03:07:43.485 INFO:teuthology.orchestra.console.smithi159:Power off completed 2024-03-19T03:07:43.593 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 908b6b276e8f14e16032b32a9770077cd37b6bd4 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1913.8222615718842 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi092 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=002a509fa8924b4d96d44ffdfcd8115e status: fail success: false 2024-03-19T03:07:43.593 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T03:07:43.686 INFO:teuthology.run:FAIL